
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based Information Extraction (IE) has tried to make an increased use of the knowledge from the target ontology in order to improve the semantic annotation results. However, only very few approaches are able to benefit from the ontology structure and one of them is not a learning system, thus is not easy to adapt to new domains, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning approach for IE, which uses the target ontology as an essential part of the extraction process. Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology.

We also carry out evaluation experiments on the largest available semantically annotated corpus of 146 classes. The results demonstrate clearly the benefits of using knowledge from the ontology for ontology-based IE. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried to make an increased use of the knowledge from the target ontology in order to improve the semantic annotation results. However, only very few approaches are able to benefit from the ontology structure and one of them is not a learning system, thus is not easy to adapt to new domains, whereas the other one does not perform semantic annotation of documents, but only ontology population.

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.  Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning algorithm for IE, which uses the target ontology as an essential part of the extraction process. The work is based on the Hieron hierarchical classification algorithm, which we extended in order to improve its performance and adapt it for Ontology-Based IE (OBIE).
This paper also presents a detailed evaluation which demonstrates clearly the benefits of using the knowledge from the ontology as one of the inputs to the IE algorithms. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population. 

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.   Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning algorithm for IE, which uses the target ontology as an essential part of the extraction process. The work is based on the Hieron hierarchical classification algorithm, which we extended in order to improve its performance and adapt it for Ontology-Based IE (OBIE).
This paper also presents a detailed evaluation which demonstrates clearly the benefits of using the knowledge from the ontology as one of the inputs to the IE algorithms. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population. 

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.   Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning algorithm for IE, which uses the target ontology as an essential part of the extraction process. The work is based on the Hieron hierarchical classification algorithm, which we extended in order to improve its performance and adapt it for Ontology-Based IE (OBIE).
This paper also presents a detailed evaluation which demonstrates clearly the benefits of using the knowledge from the ontology as one of the inputs to the IE algorithms. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population. 

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.   Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning algorithm for IE, which uses the target ontology as an essential part of the extraction process. The work is based on the Hieron hierarchical classification algorithm, which we extended in order to improve its performance and adapt it for Ontology-Based IE (OBIE).
This paper also presents a detailed evaluation which demonstrates clearly the benefits of using the knowledge from the ontology as one of the inputs to the IE algorithms. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population. 

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.   Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning algorithm for IE, which uses the target ontology as an essential part of the extraction process. The work is based on the Hieron hierarchical classification algorithm, which we extended in order to improve its performance and adapt it for Ontology-Based IE (OBIE).
This paper also presents a detailed evaluation which demonstrates clearly the benefits of using the knowledge from the ontology as one of the inputs to the IE algorithms. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population. 

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.   Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning algorithm for IE, which uses the target ontology as an essential part of the extraction process. The work is based on the Hieron hierarchical classification algorithm, which we extended in order to improve its performance and adapt it for Ontology-Based IE (OBIE).
This paper also presents a detailed evaluation which demonstrates clearly the benefits of using the knowledge from the ontology as one of the inputs to the IE algorithms. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population. 

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.   Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}
\documentclass{www2007-submission}

\begin{document}

\title{Hierarchical, Perceptron-like Learning for Ontology-Based Information Extraction}
%\subtitle{[Extended Abstract]
%\titlenote{A full version of this paper is available as
%\textit{Author's Guide to Preparing ACM SIG Proceedings Using
%\LaTeX$2_\epsilon$\ and BibTeX} at
%\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the "boxing"
% and alignment of the authors under the title, and to add
% a section for authors number 4 through n.
%
% Up to the first three authors are aligned under the title;
% use the \alignauthor commands below to handle those names
% and affiliations. Add names, affiliations, addresses for
% additional authors as the argument to \additionalauthors;
% these will be set for you without further effort on your
% part as the last section in the body of your article BEFORE
% References or any Appendices.

\numberofauthors{2}
%
% You can go ahead and credit authors number 4+ here;
% their names will appear in a section called
% "Additional Authors" just before the Appendices
% (if there are any) or Bibliography (if there
% aren't)

% Put no more than the first THREE authors in the \author command
\author{
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
%% e-mail address with \email.
\alignauthor Yaoyong Li\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{yaoyong@dcs.shef.ac.uk}
\alignauthor Kalina Bontcheva\\
	% \affaddr{Department of Computer Science}\\
       \affaddr{University of Sheffield}\\
       \affaddr{211 Portobello Street }\\
       \affaddr{Sheffield, S1 4DP, UK}\\
       \email{kalina@dcs.shef.ac.uk}
}

\maketitle
\begin{abstract}
Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population.

This paper introduces a hierarchical learning algorithm for IE, which uses the target ontology as an essential part of the extraction process. The work is based on the Hieron hierarchical classification algorithm, which we extended in order to improve its performance and adapt it for Ontology-Based IE (OBIE).
This paper also presents a detailed evaluation which demonstrates clearly the benefits of using the knowledge from the ontology as one of the inputs to the IE algorithms. We also demonstrate the advantages of our approach over other state-of-the-art learning systems on a commonly used benchmark dataset. 
\end{abstract}

% A category with the (minimum) three required fields
%\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
%\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\keywords{Ontology-based Information Extraction, semantic annotation, hierarchical learning}

\section{Introduction}

At present there is a large gap between Knowledge Management (KM) systems and the natural 
language materials that form something approaching 80\% of corporate data stores 
\cite{Per04a}. Similarly, Gartner reported in 2002\footnote{%
http://www3.gartner.com/DisplayDocument?id=379859} that for at
least the next decade more than 95\% of human-to-computer information input
will involve textual language. They also report that by 2012 taxonomic and
hierarchical knowledge mapping and indexing will be prevalent in almost all
information-rich applications. There is a tension here: between the
increasingly rich semantic models in KM systems on the one hand, and the
continuing prevalence of human language materials on the other. The process
of tying semantic models and natural language together is referred to as
{\it semantic annotation}. This process may be characterised as the dynamic
creation of interrelationships between {\it ontologies} and unstructured and semi-structured documents in a bidirectional manner. 

{\it Information Extraction} (IE), a form of natural language analysis, is becoming 
a central technology for bridging the gap between unstructured text and formal knowledge expressed in ontologies. Ontology-Based IE (OBIE) is IE which is adapted specifically for the semantic annotation task. One of the important differences between traditional IE and OBIE is in the use of a formal ontology as one of the system’s resources and may also involve reasoning. Some researchers \cite{OntoSyphon06a} call ontology-based any system which specifies its outputs with respect to an ontology, however, in our view, if a system only has a mapping between the IE outputs and the ontology, this is not sufficient and we refer to such systems as {\em ontology-oriented}. 

Another substantial difference of the semantic IE process from the traditional one is the fact that it not only finds the (most specific) class of the extracted entity, but it also identifies it, by linking it to its semantic description in the instance base. This allows entities to be traced across documents and their descriptions to be enriched during the IE process. In practical terms, this entails named entity and relation extraction and also coreference resolution both within and across documents. 

Recent work on ontology-based IE (see Section~\ref{sec:related-work}) has tried increasingly to make more and more use of the knowledge from the ontologies in order to improve the semantic annotation results. However, only very few approaches make use of the ontology structure and one of them is not a learning system, whereas the other one does not perform semantic annotation of documents, but only ontology population. 

The main contribution of this paper is to investigate the application of hierarchical classification learning for semantic annotation, as part of an ontology-based IE system.   Hierarchical classification takes into account the relations between concepts, thus benefiting directly from the ontology. In particular, this paper studies the large margin hierarchical 
classification learning algorithm Hieron proposed in \cite{Dekel04},  
because it is very efficient during both training and classification.
Computational efficiency is of major importance for OBIE, because depending on the size of the ontology, the system may need to train hundreds of classifiers. 

However, it should be noted that work presented in this paper is not a simple application of the Hieron algorithm proposed in
\cite{Dekel04} to OBIE. In fact, the OBIE requirements lead to several important modifications -- introduction of multi-loop learning and a parameter which ensures that the algorithm is applicable to any IE corpus. Both of these resulted in a quantifiable improvement in performance (see Table
\ref{Hieron-modification} in Section \ref{Sect-results}). In addition, the semantic annotation task is very different from conventional document classification, which is what Hieron was originally developed for. Consequently,
another contribution of this work is in showing how OBIE can be decomposed into  several hierarchical classification tasks, which can then be approached with an algorithm such as Hieron.

Another important contribution of this work is in the use of the Sekt\footnote{For further information on the sekt project, see http://www.sekt-project.com. The corpus itself is available on request from the second author.} ontology-annotated news corpus for detailed evaluation of our system. To the best of our knowledge, it is the only corpus suitable for evaluating OBIE, which has more than 20 classes (146 classes) from a 
non-trivial ontology (Proton). 

The problem with using only the Sekt corpus is that no other systems have been evaluated on it, apart from the SVM and Perceptron ones reported here. Unfortunately other recent corpora for IE evaluation 
(e.g., Pascal challenge\footnote{http://nlp.shef.ac.uk/pascal/Corpus.html}, 
CONLL'03\footnote{http://www.cnts.ua.ac.be/conll2003/ner/}) are either not fully available or use a small, flat set of labels (fewer than 20), thus making them inappropriate for experimenting with semantic annotation on a realistic scale.

The experimental results demonstrate that our hierarchical classification 
algorithm obtains clearly better results than SVM and Perceptron both in terms of the conventional precision and recall IE metric and an ontology-induced metric. In addition, in order to provide some comparison against state-of-the-art learning IE systems, we also evaluate our Hieron-based system on the seminar corpus, where it achieves the best overall performance. 

The paper is structured as follows. Related work on ontology-based IE is discussed in Section~\ref{sec:related-work} and our work is placed in that context. Section~\ref{obie-hieron} introduces 
the large margin hierarchical classification algorithm Hieron, our modifications
to it, the hierarchy-based cost measure, which Hieron requires, and the way in which the OBIE task is formalised as a classification problem.
The Sekt ontology-annotated gold-standard is described next (section~\ref{Sect-dataset}), 
followed by a comprehensive set of experimental results and their analysis 
(section~\ref{Sect-results}). We also compare our Hieron-based learning approach against other state-of-the-art IE learning algorithms. The paper concludes with a discussion and future work plan.   

\section{Related OBIE Systems}\label{sec:related-work}

There are a number of what we call ontology-oriented IE systems, which, unlike ontology-based ones, do not incorporate ontologies into the system, but either use them as a bridge between the IE output and the final annotation (as with AeroDAML) or rely on the user to provide the relevant information through manual annotation (as with the Amilcare-based tools). 

AeroDAML \cite{AeroDAML} applies IE techniques to automatically generate DAML annotations from web pages. it links most proper nouns and common types of relations with classes and properties in a DAML ontology. AeroDAML uses an ontology which is used to translate the extraction results into a corresponding RDF model.

Amilcare \cite{Amilcare03} is an IE system which has been integrated in several different semantic annotation tools. It uses supervised rule learning to adapt to new domains and applications given human annotated texts (training data). It treats the semantic annotations as a flat set of labels, thus ignoring the further knowledge in the ontology. Similar to us, Amilcare uses GATE's NLP components \cite{Cun02b} in order to obtain linguistic information as features for the learning process. Amilcare has formed the basis of several semantic annotation tools -- MnM \cite{MnM02} and S-CREAM \cite{Scream02} -- which combine a manual annotation tool and the IE system in the background.  

One of the problems with these annotation tools is that they do not provide the user with a way to customise the integrated language technology directly. While many users would not need or want such customisation facilities, users who already have ontologies with rich instance data will benefit if they can make this data available to the IE components. However, this is not possible when ``traditional'' IE methods like Amilcare are used, because they are not aware of the existence of the user’s ontology.

The more serious problem however, as discussed in the S-CREAM system \cite{Scream02}, is that there is often a gap between the IE output annotations and the classes and properties in the user’s ontology. The proposed solution is to write some kind of rules, such as logical rules, to resolve this. For example, an IE system would typically annotate London and UK as locations, but extra rules are needed to specify that there is a containment relationship between the two (for other examples see \cite{Scream02}). However, rule writing of the proposed kind is too difficult for most users and therefore ontology-based IE is needed, as it annotates directly with the classes and instances from the user's ontology. 

In response to these problems, a number of OBIE systems have been developed. Magpie \cite{Domingue04} is a suite of tools which supports semantic annotation of web pages. It is fully automatic and works by matching the text against instances in the ontology. The SemTag system \cite{Dill03} is similar in approach to MagPie as it annotates texts by performing lookup against the TAP ontology. It also has a second, disambiguation phase, where SemTag uses a vector-space model to assign the correct ontological class or determine that this mention does not correspond to a class in TAP. The problem with both systems is that they are not able to discover new instances and are thus restricted in terms of recall. 

The PANKOW system \cite{Pankow} exploits surface patterns and the redundancy on the Web to categorise automatically named entities found in text with respect to a given ontology. The main disadvantage of this approach is that it does not compare the context in which the proper name occurs in the document to the contexts in which it occurs on the Web, thus requiring human input in order to disambiguate instances with the same name that belong to different classes in different contexts (e.g., Niger can be a river or country). 

OntoSyphon \cite{OntoSyphon06a} is similar to PANKOW and uses the ontology as the starting point and carries out web mining to populate the ontology with instances. It uses the ontology structure to determine the relevance of the candidate instances. However, it does not carry out semantic annotation of documents, which is the problem addressed here.

The KIM OBIE system \cite{Kiryakov04} produces annotations linked both to the ontological class and to the exact individual in the instance base. For new (previously unknown) entities, new identifiers are allocated and assigned; then minimal descriptions are added to the semantic repository. KIM has a rule-based, human-engineered IE system, which uses the ontology structure during pattern matching and instance disambiguation. The only shortcoming of this approach is that it requires human intervention in order to adapt it to new ontologies.

To summarise, all these systems use the ontology as their target output and the ontology-based ones also use class and instance information during the IE process. While KIM and OntoSyphon do make use of the ontology structure, the former is a rule-based, not a learning approach, whereas the latter does not perform semantic annotation, only ontology population.  

The next section discusses how the ontology structure can be exploited, in addition to class and instance information, within a machine learning approach to ontology-based information extraction. 

\section{Exploiting The Ontology}\label{obie-hieron}

The categories of information entities in conventional IE have no specific relation among them, i.e., they are treated as 
independent of each other by the learning algorithms. 
In contrast, as concepts in an ontology are related to each other (at 
the very least through the subsumption hierarchy), it is beneficial to 
exploit this knowledge in OBIE. 
 
This section exploits  two aspects of using the ontology structure
for OBIE. First it discusses ontology-induced measures, 
which are then used by the learning algorithm, in addition to calculating some distance-based metrics. Secondly, it introduces the  
Perceptron-based learning algorithm Hieron which has
a mechanism to handle effectively hierarchical classification and our adaptations of Hieron for the OBIE requirements. 

\subsection{Ontology-Based Performance Metric\label{Sekt-measure}}

As concepts in ontologies are related to each other in a subsumption hierarchy, 
the cost (or loss) for an instance of a concept {\em X} being wrongly classified as belonging to another concept {\em Y} is defined as dependent on the two 
particular concepts (denoted as $c(X,Y)$). Given this cost measure, one OBIE requirement is that if the system misclassifies a mention in the text as belonging to class Y, instead of X, the cost of that misclassification should be as small as possible, (e.g., classify it as a super-class of the correct class). 

IE systems traditionally use evaluation metrics, 
 such as {\em precision}, {\em recall} and {\em $F_{1}$}, which are computed for 
each category independent of all other categories. An overall performance 
measure is obtained by averaging the performances for all categories 
(namely macro-averaged) or by putting together all the results of all classifications (micro-averaged). 
 However, this type of measures do take into account the hierarchical relations 
between classes in the ontology and their associated misclassification costs. Consequently, another OBIE requirement is to have performance metrics which are sensitive to the structure of the target ontology. Therefore, 
next we generalise the commonly used IE metrics, {\em precision}, {\em recall} 
and {\em $F_{1}$} to ontology-based ones.

In order to evaluate an OBIE system on a corpus annotated with a given 
ontology, we first compute the following three numbers on the system-annotated corpus:
\begin{itemize}
\item $n$ --- number of mentions which are instances of concepts in the ontology and have been found by the OBIE system (regardless of whether or not OBIE assigned them the correct class).
\item $n_{missing}$  --- number of entities in the corpus  which are instances of concepts in the ontology but are not found by the system.
\item $n_{spurious}$ --- number of the entities recognised by the system 
which actually are not an instances of any concept in the ontology.
\end{itemize}
Then each pair of concepts {\em X} and {\em Y} we define the cost measure $c(X,Y)$ as a non-negative number, equal to 0 if $X=Y$.
If we assume that $C$ is the largest cost for a given ontology, then we can 
define a cost based error as $e_{cost}(X,Y) = c(X,Y)/C$, satisfying that
$e_{cost}(X,Y)\in[0,1]$ and $e_{cost}(X,Y)=0$ if $X=Y$.

Based on the cost-based error,  an overall accuracy for the $n$ entities 
identified by the system is defined  as follows:
\begin{equation}
a_{cost} = \sum_{i=1}^{n}(1-e_{cost}(A_{i},B_{i}))
\end{equation}
where $A_{i}$ and $B_{i}$ are two classes in the ontology
and $e_{cost}(A_{i},B_{i})$ is the cost of misclassifying the $i$th entity as an instance of 
 class $B_{i}$, instead of its correct class $A_{i}$ (class
$B_{i}$ is the same as $A_{i}$ if the $i$th entity is classified correctly).

Using the overall accuracy $a_{cost}$ we define ontology-based 
precision and recall, respectively, as
\[
P_{o} = a_{cost}/(a_{cost}+n_{spurious}),   \ \ \ 
R_{o} = a_{cost}/(a_{cost}+n_{missing})
\]
Then, as with conventional f-measure, the ontology-based $F_{1}$ is defined as the harmonic mean of ontology-based precision and recall:
\begin{equation}
\label{measure-Fo1}
F_{o1} = 2*P_{o}*R_{o}/(P_{o}+R_{o})
\end{equation}

In other words, ontology-based $F_{o1}$ is a generalisation of the standard $F_{1}$. In fact, if we define the cost $c(X,Y)$ as the binary function
\begin{equation}
\label{measure-distance}
c(X,Y) = \left\{ \begin{array}{ll}
0 & \textrm{if $X=Y$}\\
1 & \textrm{otherwise}
\end{array} \right.
\end{equation}
%\end{displaymath}
then $F_{o1}$ would be reduced to the standard overall $F$-measure. 

Recent studies of hierarchical classification 
(see e.g. \cite{Dekel04,Cesa-Bianchi04}) typically define $c(X,Y)$ 
as the distance $\gamma(X,Y)$ between the two nodes $X$ and $Y$ in the classification hierarchy. In our experiments we used the distance between two classes in the ontology as their misclassification cost and henceforth this is referred to as the distance-based metric. 

\subsection{Hierarchical Learning with Hieron\label{Sect-hieron}}

The {\em Hieron} large margin learning algorithm for hierarchical classification was defined by \cite{Dekel04}, based on the margin Perceptron algorithm. 
Hierarchical classification refers to a specific 
multi-class classification problem where the class
labels are organised in a hierarchical fashion. One example  is document 
categorisation where categories belong to a taxonomy. Here 
we briefly describe the learning algorithm and discuss our modifications to the original algorithm and then discuss how to apply it to OBIE.

As defined, the Hieron algorithm exploits the 
hierarchical structure of the class labels. It learns one Perceptron model for each class and meanwhile ensures that the difference between two models is in proportion to 
the distance of the two classes in the tree. The philosophy of the learning 
algorithm is that, if we have to misclassify one example as class $X$ instead of $Y$, then that class $X$ should be close to the correct class $Y$  
in the hierarchical structure. 

Let us suppose a hierarchical classification problem which has 
instance domain $\mathcal{X}\subseteq\mathbb{R}^{n}$ and label set $\mathcal{Y}$.  
The labels in the set $\mathcal{Y}$ can be arranged as nodes in a 
rooted tree $\mathcal{T}$. For any pair of labels $u,v\in\mathcal{Y}$, let $\gamma(u,v)$
denote their distance in the tree, namely the number of edges along the (unique)
path from $u$ to $v$ in $\mathcal{T}$. For every label $v$ in the tree, $\mathcal{P}(v)$ 
is defined as the set of labels along the path from the root to $v$, inclusive.

The training set $\mathcal{S}=\{(\mathbf{x}_{i},y_{i}): i=1,\ldots,m\}$ 
contains instance-label pairs, where  each  $\mathbf{x}_{i}\in\mathcal{X}$ and each 
$y_{i}\in\mathcal{Y}$. The Hieron learning algorithm  
aims to learn a classification function $f:\mathcal{X}\to\mathcal{Y}$ which has a 
small tree induced error. The classifier $f$ has the following form: each label 
$v\in\mathcal{Y}$ has a matching prototype $\mathbf{W}^{v}\in\mathbb{R}^{n}$, 
and the classifier $f$ makes its predictions according to the following rule:
\begin{equation}
f(\mathbf{x}) = \mathrm{argmax}_{v\in\mathcal{Y}}\langle\mathbf{W}^{v},\mathbf{x}\rangle
\end{equation}
where $\langle\cdot,\cdot\rangle$ represents the inner product of two vectors. 
Hence, the task of learning $f$ is reduced to learning a set of prototypes 
$\{\mathbf{W}^{v}:v\in\mathcal{Y}\}$. 

However, Hieron does not deal directly with the set of prototypes but rather 
with the difference between the prototype of each node and the prototype of its parent. 
Formally, we denote $\mathcal{A}(v)$ as the parent node of $v$ in the tree
and assume that the parent node of a root node is the root itself.
Then the difference weight vector is defined as
$\mathbf{w}^{v} = \mathbf{W}^{v} - \mathbf{W}^{\mathcal{A}(v)}$. 
Each prototype is now decomposed into the sum
\begin{equation}
\label{Eqn-prototype}
\mathbf{W}^{v} = \sum_{u\in\mathcal{P}(v)}\mathbf{w}^{u}
\end{equation}

Since the learning algorithm requires that adjacent vertices in the label tree 
have similar prototypes, by representing each prototype as a sum of vectors 
from $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$, adjacent prototypes $\mathbf{W}^{v}$ 
and $\mathbf{W}^{\mathcal{A}(v)}$ can be kept close by simply keeping the norm of the
weight vector $\mathbf{w}^{v}=\mathbf{W}^{v} -\mathbf{W}^{\mathcal{A}(v)}$ small.

The Hieron algorithm assumes that there exists a set of weight vectors 
$\{\boldsymbol{\omega}^{v}:v\in\mathcal{Y}\}$
such that the following hold:
\begin{eqnarray}
\label{eqn-assum}
\sum_{v\in\mathcal{P}(y_{i})}\langle\boldsymbol{\omega}^{v},\mathbf{x}_{i}\rangle - 
\sum_{u\in\mathcal{P}(r)}\langle\boldsymbol{\omega}^{u},\mathbf{x}_{i}\rangle \ge 
\sqrt{\gamma(y_{i},r)}, \\
\ \forall(\mathbf{x}_{i},y_{i})\in\mathcal{S}
 \ \mathrm{and} \ \forall r\in\mathcal{Y}\backslash\{y_{i}\} \nonumber
\end{eqnarray}
%The  difference in (\ref{eqn-assum}) is a generalisation of the notion of margin 
%employed by multi-class problems for hierarchical classification 
%(see \cite{Dekel04} for details). 
However, note that this assumption can be relaxed if we 
introduce some regulation parameter into the learning algorithm, as discussed below.

The Hieron learning algorithm is similar to the Perceptron algorithm as
it learns one classifier per class. But, unlike Perceptron where  
each model is learned independently of the others, it learns the 
Perceptrons for all classes in a collective way. 
The algorithm 
initialises each of the Perceptron's weight vectors $\{\mathbf{w}^{v}:v\in\mathcal{Y}\}$ 
as a zero vector and updates a weight vector only if a prototype related with 
it makes a wrong prediction. By doing so the learning algorithm tries to keep 
the norm of the weight vector small, which is one of the requirements as discussed above. 

The learning algorithm also tries to satisfy the margins requirement for the 
weight vectors and training set shown in (\ref{eqn-assum}). Formally, for each 
instance-label pair $(\mathbf{x}_{i},y_{i})\in\mathcal{S}$, the learning 
algorithm checks if the current weight vectors satisfy the 
margin requirement for each label $y\ne y_{i}$ by computing the following 
loss function:
\begin{eqnarray}
\label{eqn-loss}
L(\{\mathbf{w}^{v}\},\mathbf{x}_{i}, y_{i}, y) & =& 
\sum_{u\in\mathcal{P}(y)}\langle\mathbf{w}^{u},\mathbf{x}_{i}\rangle 
-\sum_{v\in\mathcal{P}(y_{i})}\langle\mathbf{w}^{v},\mathbf{x}_{i}\rangle \nonumber \\
 & & + \sqrt{\gamma(y_{i},y)}
\end{eqnarray}
The margin requirement for $(\mathbf{x}_{i},y_{i})$ and $y$ is satisfied if 
and only if the above function is less than or equal to 
$0$. If the margin requirement is satisfied for all training examples, then the 
learning stops and returns the current learned model. Otherwise, from all 
training examples $(\mathbf{x}_{i},y_{i})$ for which the margin requirement 
(\ref{eqn-assum}) is violated by the current model, 
choose the label $\hat{y}_{i}$ that violate the margin requirement the most 
(namely it has the maximal value of the function (\ref{eqn-loss})), 
and update the current weight vectors comprising the two prototypes $\mathbf{W}^{y_{i}}$ and 
$\mathbf{W}^{\hat{y}_{i}}$, respectively, as illustrated in Figure \ref{Fig-update}.

As shown in Figure \ref{Fig-update}, when a training 
example $\mathbf{x}$ with label $y$ is predicted mistakenly as label 
$y'$, only the weight vectors
associated with the nodes in the shortest path linking nodes $y$ and  
$y'$ are updated, except for the MSCA node. In other words, only
the nodes depicted using solid lines are updated, in which the symbol '+' 
means increasing the corresponding weight vector by the example $\mathbf{x}$ 
and the symbol '-' means decreasing the weight vector by $\mathbf{x}$.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth,height = 0.18\textheight ]{../images/graph-update-hieron} \ \
\end{center}
\vspace{-0.6cm} \caption{Illustration of the Hieron's update.}
\label{Fig-update}
\end{figure}

As already discussed above, in order to ensure that adjacent vertices in the 
label tree have similar prototypes, the Hieron algorithm needs to keep the norms 
of the weight vector $\mathbf{w}$ as 
small as possible. This is achieved by initialising all weight vectors to zero
and only updating them if necessary.

\subsection{Our Hieron Modifications}

The learning algorithm described above is the original 
Hieron batch learning algorithm presented in \cite{Dekel04}. In order to achieve
better performance,  some 
modifications were introduced in our implementation, as follows: 

\begin{enumerate}
\item Our learning algorithm  learns from the training set until no error is 
made on the training examples, which means that more than one learning loop may 
be needed on the training set. In contrast, the original
Hieron batch learning just allowed one learning loop. It 
will be shown by our experiments described below that multi-loop learning has 
better generalisation performance than single loop learning.

\item The Hieron learning algorithm requires that the training set is compatible
 with the margin conditions described in equation (\ref{eqn-assum}), so that the algorithms stops after a finite number of loops.  
This is a problem because in OBIE often it is not known in advance whether or not a training set satisfies the margin condition. Therefore, we introduce a 
regulation parameter into the algorithm such that the learning would stop after 
some loops on any training set. The value of the regulation parameter 
is a positive number\footnote{Although the optimal value of regulation parameter may be dependent
on the data used, in our experiments described in this paper, we simply set
the regulation parameter as 1.}. The regulation parameter is similar to that used 
for Perceptron (see e.g. \cite{Yaoyong02a}). In more detail, the role of the regulation parameter is to add an extra dimension to the
feature vector of each training example and set the component of the dimension
as the regulation parameter. By doing so the set of the training examples with
extended feature vectors would become linearly separable regardless of the linear separability of the original training set, meaning that the Hieron algorithm would stop after
a finite number of loops for any training data.
\end{enumerate}

In addition, the original algorithm \cite{Dekel04} distinguishes two types of learning models. 
One type was the weight vectors obtained at the end of learning, namely 
$\{\mathbf{w}_{t}^{m}: v\in\mathcal{Y}\}$. Another was the mean of all weight vectors 
used during learning. Let us assume that we apply the weight vectors $m$ times 
to training examples during learning and the weight vectors used  were 
$\{\mathbf{w}_{i}^{v}: v\in\mathcal{Y}, i=1,\cdots,m\}$, then for every $v\in\mathcal{Y}$ 
define the means of weight vectors as
\begin{equation}
\mathbf{w}^{v} = \frac{1}{m}\sum_{i=1}^{m}\mathbf{w}_{i}^{v}
\end{equation}
It was shown in \cite{Dekel04} that the averaged weight vectors had better 
results than the last weight vectors in most cases. In our OBIE experiments we also compare the two types of weight vectors (see Section~\ref{Sect-results}).

\subsection{The Hieron-based OBIE System}

As already discussed, the goal of OBIE is to identity and classify information 
entities in text as instances of concepts in an ontology. On the other hand, 
the Hieron is basically a classification algorithm which classifies every 
example into categories organised in a tree structure. In order to apply the 
Hieron to OBIE, we need to adapt the OBIE task to make it similar to the 
hierarchical classification problem. 

First, we convert the OBIE task into two hierarchical classification problems. 
Traditionally, when classifiers are used for information extraction each token in the text is treated as one example and classified as belonging to one of the target IE labels or as having no label (see e.g., \cite{Fre00,Yaoyong05a}). In particular, 
the annotation task can be viewed as two binary classification 
problems, one is for recognising the start tokens of information entities 
and the other one is for the end tokens. Similarly,
we transform the OBIE task into two hierarchical classification problems. 
For each class in the ontology, two hierarchical classifiers are trained -- one for 
recognising the start token of the class instances and one for the end.  

Secondly, as there are tokens in the text which do not belong to any class in the ontology, in order to apply Hieron to OBIE,
 the ontology is extended notionally with a new child node of the root node, that represents the concept of non-relevant token. However, this added 
concept is not considered in the evaluation metrics, as it is only required for the proper functioning of the classification algorithm.

Thirdly, note that the Hieron algorithm requires that classes are organised 
in a tree. However, for some ontologies, the class subsumption hierarchy 
is not a tree, but a graph where some concepts occur as subclasses of two or more classes. In other words, some nodes in the ontology may have more than one parent.
The Proton ontology used in our experiments (see below) is one example of this kind
of ontology. For example, the concept {\em SportBuilding} is a sub-concept
of  {\em Building} and {\em SportFacility}.

Consequently, 
the Hieron algorithm had to be adapted to deal with graph-like hierarchies, such as the Proton ontology. 
The modification we made on Hieron was simple. We compute one prototype vector 
for each path from the root node to the given class using formula 
(\ref{Eqn-prototype}). Then, given one
example during training or application, the inner products between that example and every prototype
vectors are compared with each other and the example is assigned the class whose prototype is most relevant.

%The training phase of the Hieron algorithm did not require any changes, 
%because it only considers the shortest path between a pair of nodes which can be 
%obtained unambiguously even in graphs. was made to the 
%application phase where  rather 
%than only one prototype for a node in the case of trees. 

Finally, we replace the distance $\gamma(X,Y)$ in the Hieron algorithm with the 
cost measure $c(X,Y)$ between two concepts in the ontology. Therefore, we 
can learn classifiers which are optimised according to a distance-based cost measure, which encodes structural knowledge from the ontology. 

\section{Experimental Dataset}
\label{Sect-dataset}
The corpus used in our experiments consists of semantically annotated news articles. The articles
were divided into three subsets according to the article's theme, namely business, 
international politics and UK politics, 
which has 91, 99 and 100 articles, respectively.
The corpus was annotated manually according to
the Proton ontology\footnote{See http://proton.semanticweb.org.}.
Our experiments described below used the {\em psys:Entity} branch of the Proton ontology that has 272 unique concepts. 
%Figure \ref{fig:ontology:proton} presents a part of 
%the Proton ontology. This ontology section
%starts with the root node {\em Thing}, and has 10 levels of concepts
%with {\em TVCompany} as the lowest level concept. Note that the figure does not
%show all child concepts for most classes. 

%\begin{figure}[htbp]
%\begin{center}
%\includegraphics[scale=0.6]{../images/proton-part}
%\caption{Subset of the Proton ontology} \label{fig:ontology:proton}
%\end{center}
%\end{figure}

As already discussed above, some concepts in Proton have more than one parent 
class and the 
hierarchical structure of Proton has 10 levels, with maximal path length 14. 

The news corpus was annotated with 167 concepts, of which 146 concepts are in the 
Proton ontology and are used  in our experiments\footnote{The other 21 concepts 
were proposals as extensions to the ontology, but have not been adopted yet and 
therefore are not used in these experiments.}.
The concepts span from the 3rd to the 10th level of the hierarchical 
structure of Proton. As the corpus was created in the Sekt project, hereafter we 
will refer to the corpus as the Sekt ontology-annotated news corpus. 

\begin{table}[htb]
\caption{Distribution of concepts with different numbers of instances in the Sekt 
ontology-annotated corpus. \label{Distribution-onto}}
{\centering
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline
\#instances & 1 &  2 & 3 & 4  & 5 & 6 -- 10  & 11 -- 20   & $>$20  \\
\hline
\#concepts  & 23 & 14 & 12 & 6 & 2 & 13 & 21 & 55\\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Distribution-onto} presents the distribution of concepts with different numbers of instances in the corpus. We can see that there are 57 concepts each of which has 5 instances or less, thus presenting a data sparseness problem.

\begin{table*}[ht]
\caption{Numbers of instances of the seven concepts in the three parts of the 
Sekt ontology-annotated corpus. \label{Statistics-news-onto}}
{\centering
\begin{tabular}{lcccccccc} \hline
         & \#Doc &  Person & Loc & Org  & Money  & Contact  & Temporal & Time  \\
\hline
Business & 91 & 336 & 601 & 1385 & 490 & 2 & 119 & 743\\
International      & 99 & 948 & 1857 & 840 & 75 & 1 & 112 & 550\\
UK      &100 & 845 & 847 & 845 & 100& 0 & 106 & 670 \\
 \hline
\end{tabular}\par}
\end{table*}

\begin{table*}[htb]
\caption{Example features for the text ``Time: 3:30 PM''. The Unknown value for the word ``Time'' means that ANNIE identified
the word ``Time'' as a named entity but could not attribute a more specific entity type. 
\label{Feature_text}} \vspace{0.2cm} {\centering
\begin{tabular}{r|c|c|c|c|c|c} \hline
    Token & Case & Tokenkind & Lemma & Pos & Lookup & ANNIE Entity \\
\hline
Time& upperInitial & word & time & NNP & & Unknown\\
%\hline
 : &   & punctuation & : & : & &  \\
3 &  & number & 3 & CD & & Time \\
: &   & punctuation & : & : & & Time \\
30  &  & number & 3 & CD & & Time  \\
PM & allCaps & word & pm & NNP & time & Time\\

 \hline

\end{tabular}\par}
\end{table*}

In order to examine the effect of data sparseness on the algorithm's performance, 
we also created a version of the corpus where all classes were generalised into 
7 high-level classes, which are broadly equivalent to labels used in traditional 
IE systems (e.g., 
Person, Location, etc). Table \ref{Statistics-news-onto} presents the numbers of 
instance of each of the seven concepts in each part of the corpus.

The corpus is pre-processed with the open-source ANNIE
system, which is part of GATE \cite{Cun02b}. This enabled us to
use a number of linguistic features, in
addition to information already present in the document such as words
and capitalisation information. The linguistic features are
domain-independent and include token kind (word, number,
punctuation), lemma, part-of-speech (POS) tag, gazetteer class, and
named entity type according to ANNIE's rule-based name entity recogniser\footnote{These 
entity labels are: Person, Organisation, Location, Date, Money, and Address.}.  
Table \ref{Feature_text} shows an example of  text with its associated  features. Note that a token may not have values for all features, e.g. the token ``Time'' does not have value for the Lookup feature because it does not occur in  ANNIE's gazetteer lists.

Since in IE the context 
of the token is usually as important as the token itself, the learning algorithm takes into account features of the 
preceding and 
following tokens, in addition to those of the token being classified. In our
experiments the same number of left and right tokens was taken as a
context window. In the experiments 
discussed below, the window size is 4, which means that the algorithm uses features 
derived from 9 tokens: the
4 preceding, the current, and the 4 following tokens.
Due to the use of a context window, the input vector is the
combination of the feature vector of the current token and these of
its neighboring tokens. 

\section{Experimental Results}
\label{Sect-results}
The experiments reported here evaluate our modified implementation of the Hieron learning algorithm on the Sekt ontology-annotated news corpus. As there are no previously reported results on this corpus, we compare Hieron against two state-of-the-art ``traditional'' learning algorithms for IE: SVM and Perceptron. 
Since Hieron exploits the relationships among classes in the ontology, the 
expectation is that it would perform better on OBIE than SVM and Perceptron which were designed basically for flat classification. In other words, these algorithms ignore the relationships between the classes and 
treat them as independent of each other. 

As already discussed in section~\ref{Sect-hieron}, the Hieron algorithm
is very similar to the uneven margins Perceptron except that Hieron takes 
into account the relationship among classes as well as the misclassification cost measure $c(X,Y)$.

In these experiments, we used the uneven margins SVM and Perceptron with uneven  
margins (PAUM), instead of the standard algorithms, because the uneven 
margins algorithms have better performance than the respective 
standard models for IE (see \cite{Yaoyong05a}).  

\begin{table}[ht]
\caption{Comparison of the performance of Hieron, SVM and Perceptron learning on OBIE: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$(\%).
% which is based on the distance between labels.
%PAUM refers to a variant of Perceptron learning, Perceptron Algorithm with uneven margins. 
\label{Results-hieron} }
{\centering
\begin{tabular}{l|ccc|ccc}   
\hline  
& \multicolumn{3}{l}{Micro-averaged  $F_{1}$} &
 \multicolumn{3}{l}{Distance induced $F_{o1}$}\\
\hline
& PAUM & SVM & Hieron&  PAUM & SVM & Hieron\\
\hline
Bu.      &74.1 & 75.3 & {\bf 82.7} &78.8 & 79.3 &{\bf 91.2} \\
Int      & 77.1& 80.1 & {\bf 83.3} &83.0 &85.9 &{\bf 91.3} \\
 UK      &82.0& {\bf 82.9}&  82.5 &83.6 &84.4 &{\bf 90.1}\\
\hline
\end{tabular}}
\end{table}

Table \ref{Results-hieron} presents the results of the three learning 
algorithms on the Sekt ontology-annotated news corpus, measured both by conventional micro-averaged $F_{1}$ and the distance-based $F_{o1}$ defined in formula 
(\ref{measure-Fo1}). Recall that the micro-averaged f-measure treats concepts in ontology as independent, whereas the distance-based metric takes into account the relationships in the ontology. In other words, if a mention in the text is found, but assigned mistakenly the wrong ontology class, then no credit is given with the conventional micro-averaged F1, but some credit (which is inversely proportional to the distance between the two concepts in ontology) will be given by the distance-based measure.

For each algorithm, we run three experiments which use one
of the three subsets of the corpus for testing and the other two for training.

Firstly, as shown in Table~\ref{Results-hieron}, Hieron outperforms the non-ontology-based IE methods measured by conventional $F_{1}$. 
The 5 to 10\% improvement in results demonstrates that the IE algorithm does benefit from taking into account the 
relationships between classes in the ontology and using this information during 
the learning process to optimise performance.  

In addition, 
Hieron consistently achieves significantly higher performance  than 
SVM and Perceptron according to the distance-based $F_{o1}$, which gives partial credit for ontological ``near misses''.
This increased performance is due to the optimisation mechanism built
into Hieron, where the distance between the classifiers for
two concepts is proportional to the distance of the two concepts in the ontology. 
In contrast, SVM and other traditional IE learning methods are trained
to treat each concept independently from the rest. Consequently, when Hieron misclassifies a mention, it is much more likely than the flat algorithms to assign a close concept if it cannot identify the correct one exactly
Consequently, this is the reason for the big difference in performance between the two metrics. 

%Secondly, 
%despite that the Hieron recognised more entities than the SVM (as shown
%by the higher conventional $F1$), it 
%the results of distance-based $F_{o1}$ show that even when 
%misclassifying tokens, 

Table \ref{Time-hieron} compares the computation times of training and
application of the three learning algorithms on the ontology corpus. The
business and international politics parts of the corpus were used for
training and the UK politics part for testing. We ran those experiments
on a Linux server with one Pentium 4 CPU (3.0GHz) and 2G RAM. 
The results show that Perceptron is very fast, whereas SVM takes much longer 
than both Perceptron and Hieron, particularly for application. 
At the same time, its performance in all experiment is only slightly better than Perceptron. 

The Hieron algorithm takes longer than Perceptron, but performs significantly
better than both traditional IE algorithms. As the test set consisted of 100 documents, the Hieron effectively took on average 1 second per document. 

\begin{table}[htb]
\caption{Training and test times (in second) of the learning algorithms: Perceptron, SVM, 
and Hieron.
\label{Time-hieron}}  
{\centering
\begin{tabular}{lrrr}
\hline
& PAUM & SVM & Hieron\\
\hline
Training &552s & 20528s &3815s\\
Test    & 33s  & 11187s &109s\\
\hline
\end{tabular}\par}
\end{table}

\begin{table}[htb]
\caption{Comparisons of the three variants of the Hieron: 
micro-averaged $F_{1}$ (\%) and distance induced $F_{o1}$ (\%). 
 Training time is shown in second.
\label{Hieron-modification}}
{\centering
\begin{tabular}{lcccccc}  
\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
\hline
& Last & Mean & Last & Mean & Last & Mean\\
\hline
MA  & 79.8 & 79.1  &81.3 & 82.2& {\bf 82.5} & {\bf 82.5}\\
Dist  &89.0 &88.3   &89.3 & 89.8& {\bf 90.1}  & 90.0\\
time &510s  & 989s &54173s & 97460s  & 4124s& 11416s\\
\hline
%\begin{tabular}{lccc}  
%\hline &\multicolumn{2}{l}{Single loop}& \multicolumn{2}{l}{Multi-loop} &\multicolumn{2}{l}{Regulation}\\
%\hline & Single loop & Multi-loop & Regulation\\
%Micro-averaged  $F_{1}$ & 85.2  &86.7  & 87.9 \\
%Distance induced $F_{1}$ &89.0    &89.3  & 90.1 \\
%Training time &516s  &54173s   & 3815s \\
%\hline
\end{tabular}}
\end{table}

As already discussed in Section \ref{Sect-hieron}, we modified the Hieron algorithm  presented in \cite{Dekel04} by introducing multiple 
learning cycles on the training set. We also introduced a regulation parameter to each weight vector to guarantee that the training would finish after a finite number of learning cycles on any training corpus. 
Table \ref{Hieron-modification} compares the performance of the original Hieron version against our modified algorithm, using the business and international politics subsets for training and the UK politics subset for testing.  
Results for the last weight as well as the mean of all obtained weights during 
learning are reported for all three Hieron variants.

As shown in Table \ref{Hieron-modification}, 
multi-loop learning (300 loops used in
the experiment) on its own achieves 
better results than single loop learning, in particular with respect to
the conventional micro-averaged $F_{1}$, showing that multi-loop learning can exploit better the data regularity than the single loop can. 

Secondly, when the regulation parameter is added 
to the multi-loop learning the performance improves even further, while the 
training time of the former is only about one fourteenth of
 the time of the latter. 
 
Finally, averaged weights perform slightly better than 
the last weight in some cases and a bit worse in the other cases. However, this also requires much higher computation time than the last weights, so there is a trade-off between performance and training time which means that for practical systems using last weight would probably be more useful. 

Last but not least, it should be noted that the differences 
in computation times between these variants of the Hieron algorithm affect only training and application times are the same. 

\subsection{Scoring Partially Correct Results \label{sect-match}}

When IE systems are evaluated the entities predicted by the system are
compared to the entities in the human-annotated gold standard. In the above experiments we used Hieron to 
recognise separately the start and end tokens of the instances. Therefore, if
both the start and end boundaries are recognised correctly, we 
say the entity is recognised correctly or is an exact match to the gold standard. On the other hand, if only the start or end token match those from the gold standard, then it is referred to as a partial match or a partially correct result.

Few IE systems (e.g. \cite{Ciravegna01b}) are
evaluated by using exact match only and partially correct results are discarded. 
However, traditional large-scale evaluations of IE systems, e.g., the Message Understanding Conferences (MUC) give also credit to partial matches \cite{Chinchor92}. Consequently, in the experiments reported above, we took into account both correct and partially correct results and, again following traditional IE scoring methods, partial results are assigned half the score of an exact match. 

Table \ref{Hieron-match} compares the results for exact match against those for correct and partial match combined. For exact match Hieron performs worse
than SVM with respect to conventional f-measure. However,
if partially correct results  are taken into account, Hieron's results are higher, showing that the hierarchical classifier has better capability than the ``flat'' SVM IE in recognising the presence of instances in the text, even when it is not able to recognise their boundaries exactly.

\begin{table}[tb]
\caption{Comparison of the exact match and partial match for the SVM and Hieron: 
conventional micro-averaged $F_{1}$ (\%) and the distance based $F_{o1}$ (\%).
\label{Hieron-match}}
 {\centering
\begin{tabular}{l|cc|cc}  
\hline
  & \multicolumn{2}{c}{ $F_{1}$} & \multicolumn{2}{c}{Distance  $F_{o1}$}\\
\hline
                 & SVM & Hieron & SVM & Hieron \\
\hline
 Exact            & 79.3 & 76.7 & 80.5 &82.0 \\
 Exact and partial &  82.9 & 82.5 & 84.4 &90.1 \\
\hline
\end{tabular}\par}
\end{table}

\subsection{Heterogeneous and Homogenous Data}

As already discussed above, the Sekt ontology annotated corpus contains documents from three different news types, with around 90 documents of each type. In the experiments discussed
above, we used two of these parts for training and the third one for testing, resulting in heterogeneous training and test data. In addition, we ran an experiment where
two third of the documents from each three parts were put together as training data and the remaining documents were used 
for testing. Consequently the training and testing data in this case are homogenous. 

Using 3-fold cross-validation, the mean results were:\\
conventional $F_{1}$ = 0.850;
distance based $F_{1}$ = 0.932; 
Compared to the results of heterogeneous training and testing data listed in Table
\ref{Results-hieron}, the homogenous training and testing data helped us achieve better results, but not as much as expected. We attribute this to the fact that the documents in the corpus are annotated with classes from a general ontology (Proton), which does not contain domain 
specific concepts which only occur in one or two of the subsets.

\subsection{Learning curves}

In order to establish the effect of data sparseness on the system's performance,
 we carried out experiments to compare the learning curves for the top 7 
classes (listed in Table \ref{Statistics-news-onto}) with those for all  
classes in the Sekt ontology annotated corpus. 
%As shown in Figure \ref{fig:ontology:proton} 
The top 7 classes are high level concepts
in the Proton ontology and have sub-concepts. For example, {\em Organisation} subsumes 
{\em CommercialOrganisation}, 
{\em EducationOrganisation} and {\em PoliticalEntity}, etc. Figure \ref{hieron-adaptive}
shows the learning curves, using standard IE micro-averaged $F_{1}$ on, respectively, the top 7 classes and 
all classes. Each experiment
constituted of ten runs and in each run $N$ documents were selected at random from the Sekt
ontology annotated corpus\footnote{Each run we selected the same number of  documents from each of the three parts of the corpus for training. In other words, the training and test data was homogenous.}
as training data and all other documents for testing. 
The results for each experiment are the means over the ten runs, with the confidence intervals at the 95\%  level.

The results show, unsurprisingly, that
the more training documents are used, the better the results are. Secondly,
on small training sets (10 or 20 documents), the results for the 7 classes are 
much better than for all classes. This is due to data sparseness, as each document 
contains many more examples when the class labels are generalised to the top 7. 
The training time for 7 classes is also much
less than that for all classes, due to the need for training fewer classifiers. Hence, if detailed information is not required, then
learning the high level concepts is better, as it needs less training data and 
is also much faster.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-7classes-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Hieron performance on different amounts of training data for the top 7 classes and all classes respectively}
\label{hieron-adaptive}
\end{figure}


In addition, it is also interesting to compare the different performance measures 
as well as the different learning algorithms on small training sets, because in many 
applications there is often only a small amount of annotated training data.
Figure \ref{hieron-adaptive-measures} presents the results of Hieron, SVM and PAUM for all classes on the Sekt ontology annotated corpus, measured both with conventional and distance-based $F_{1}$. 

Firstly, for Hieron the difference between 
the conventional measure and the ontology induced measure
is much bigger on small training data than on a larger number of documents. 
On the other hand, for SVM (and PAUM) the differences between the different measures changed slowly with respect to the size of the dataset. 
This reflects the different learning mechanisms adopted by Hieron and SVM, respectively. 
While the only aim of SVM is to classify every instance correctly, the Hieron classifier at first tries to classify an instance as correctly as it possibly can, and if it cannot classify the instance correctly, it then tries to classify it at the lowest possible
cost, relative to the correct concept. Therefore, given a sufficiently large training set, 
Hieron can learn a good classifier for each class and when the learned model 
is applied it would classify many instance correctly and minimise the cost on a small number of incorrectly
classified ones. This leads to a small difference between the conventional
measure which considers classes separately and the ontology-based measure which takes into account the relations between classes. 

In contrast, if using a small training set,  Hieron might
not be able to learn a good classifier for some individual classes due to data sparsity. So when this model is applied,
it would classify some examples correctly and minimise the cost on many of the misclassified 
instances. Consequently the ontology-based measure is much higher than
the conventional one for small training sets.

Secondly, Hieron has significantly better results than SVM on almost every training set. The ontology-based measures for Hieron are much higher those for SVM, which in turn performs better than PAUM. 
On the other hand, PAUM is much faster for training and testing than both
the SVM and Hieron. The SVM took much longer time than Hieron,
in particular for testing. Hence, overall we can conclude that Hieron is a good learning algorithm for OBIE, because it balances performance and computational complexity.

\begin{figure}[htb]
\begin{center} 
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-conv-lc} \ \
\includegraphics[width = 0.38\textwidth]{../images/www07-comparison-distance-lc} \ \
\end{center}
\vspace{-0.6cm} \caption{Comparison of Hieron, SVM and PAUM performance on small training sets}
\label{hieron-adaptive-measures}
\end{figure}

\section{Evaluation against Other Methods}
\label{sect-ie-comp}

Since are no existing publically available corpora\footnote{The most recent Pascal evaluation challenge for IE systems has not yet released its human-annorated test data, so cannot be used to compare against other systems.}, annotated with more than 10 classes, we evaluated our Hieron-based system on the recently created Sekt corpus. Unfortunately, this makes it difficult to compare our approach to previous systems, as this corpus is new and no previous evaluations have been reported on it, except those reported here. 

On the other hand, there exist several benchmarking corpora on which many IE systems have been evaluated. In order to make a comparison of our Hieron-based learning algorithm to other state of the art systems for IE, we applied our algorithm to the CMU Seminars corpus, which standard benchmarking data for conventional IE.

The Seminars corpus has been used for evaluating many IE systems. This includes
rule learning based system such as
Rapier \cite{Califf98}, BWI
\cite{Freitag&Kushmerick'2000}, SNoW \cite{Roth01} and $(LP)^{2}$
\cite{Ciravegna01b}, as well as statistical learning systems such as
HMM \cite{Freigtag&McCallum'99} and maximum entropy (MaxEnt) \cite{Chieu03}.

The Seminars corpus contains 485 Seminar announcement posts and is annotated
with four types of information entities, namely start time (stime),
end time (etime), speaker and location of one seminar. 

In order to make a fair
comparison with other state-of-the-art learning algorithms for IE,
our experiment settings followed the methodology of Rapier and $(LP)^{2}$. First, the results are the average over 10 runs and in each run 243 documents are randomly selected from the corpus as training data and the remaining 242 are used for testing. 

Secondly, as far as possible, we used the same features as the other systems used to enable a more informative comparison. In particular, the results discussed here, including our system, do not use any gazetteer information and named entity recogniser output. The only features used in this case are words,
capitalisation information, token types, lemmas, and POS tags. Finally, we
use the exact match for evaluating the results and do not consider
partially correct annotations (see Section \ref{sect-match}), because the results of the other systems only considered the exact match as well. In other words, an entity is regarded as being identified correctly only if both its start and end tokens are recognised correctly.

Since Hieron needs relations among the entity types, in order to apply
it to this corpus, we constructed a trivial ontology which
consists of one concept {\em Thing} as a root node,
{\em Entity} as the only child of {\em Thing}, and the four entity types {\em Stime}, {\em Etime}, {\em Speaker} and {\em Location} as the child concepts of {\em Entity}. 

\begin{table}[tb]

\caption{Comparison of our Hieron-based system to other learning algorithms on the seminar corpus: $F_{1}$ (\%)
on each slot and macro-averaged $F_{1}$.  The 95\% confidence interval
for MA $F_{1}$ is presented for Hieron. The best results for each
slot and for overall performance appear in bold.
\label{Comparison-seminar}} \vspace{0.2cm} 
{\centering
\begin{tabular}{lccccl}   \hline  & speaker & location & stime
 & etime & MA $F_{1}$ \\
\hline
     Hieron   & 75.5 & 83.1 & 95.4 & 93.7 & {\bf 87.0{\small $\pm1.1$}}\\
    SVM    & 69.0 & 81.3 & 94.8 & 92.7 & 84.5 \\
    $(LP)^{2}$ & {\bf 77.6} & 75.0 & 99.0 & 95.5&  86.8\\
    SNoW       & 73.8 & 75.2 & {\bf 99.6} & {\bf 96.3}& 86.2  \\
    MaxEnt    & 65.3 & 82.3  & {\bf 99.6}  & 94.5 & 85.4 \\
    BWI       & 67.7 & 76.7 & {\bf 99.6} & 93.9& 84.6    \\
    HMM       & 71.1 & {\bf 83.9} & 99.1 &59.5& 78.4   \\
    Rapier   & 53.1 & 73.4 & 95.9 & 94.6& 79.1    \\
 \hline
\end{tabular}\par}
\end{table}

Table \ref{Comparison-seminar} presents the results of our Hieron-based algorithm compared to other IE systems on the Seminars corpus. The SVM results are from our earlier experiments with uneven margins SVM on this corpus \cite{Yaoyong05a}. The results of the other systems were obtained from the papers cited at the start of this section. 

The results show that on each of the four slots our system's performance is not far from the best per-slot measures which are achieved by different systems.
As a result, our Hieron-based system obtained the best overall performance, although its performance is not significantly better from the next two systems: $(LP)^{2}$ and SNoW. From this, we can conclude that our algorithm is also a very good, state-of-the-art learning algorithm for information extraction.

In our view, the excellent perfomance of our Hieron-based system is due to its exploiting the trivial seminar ontology which we created to meet the algorithm's requirements. In this ontology, the four classes of interest have the same distance to each other, whereas the distance is larger between each of them and the non-entity concept, which we added to the ontology as a child concept of the top concept {\em Thing}. Consequently, due to Hieron's learning mechanism, the classifier for the non-entity class is more different than the classifiers for these 4 target classes. 
This leads to Hieron making fewer misclassifications than the
other learning algorithms (e.g., SVM), which do not explore the asymmetric similarities between the classes and non-entities in the Seminar corpus. 

\section{Conclusion}

This paper investigated the adaptation and evaluation on ontology-based information extraction of a large margin hierarchical classification, Perceptron-like algorithm Hieron. The algorithm takes 
into account the relations among concepts, thus benefiting directly from the 
ontology structure. We made several modifications
to the original Hieron algorithm presented in \cite{Dekel04}, which, as proven
by our evaluation, led to an improved performance.

The algorithm's performance is evaluated on the biggest available semantically annotated corpus, covering 146 concepts from the Proton ontology. Our system is compared to SVM and Perceptron, which are two state-of-the-art learning algorithms for IE. The results showed that our hierarchical 
classification algorithm obtained significantly better results than Perceptron and SVM. 

The approach was evaluated on the Sekt ontology-annotated news corpus, as it is the only corpus for the OBIE which is annotated with more than 20 classes from a  
non-trivial ontology. The problem is that no other systems have been evaluated 
on it, apart from the SVM and Perceptron comparisons reported here. Unfortunately other recent corpora for IE evaluation 
use a flat set of labels, thus making them inappropriate for OBIE. However, in order to enable some comparison of our approach to other work, we also ran Hieron on the 
Seminar corpus, despite it being a flat set of labels. The experimental results showed that our system outperformed again other state-of-the-art learning algorithms, proving that Hieron is well suited for information extraction tasks.

Further work will focus on integrating active learning with the Hieron algorithm and further evaluation on related tasks, e.g., opinion mining. 

\bibliographystyle{abbrv}
%\bibliography{../big}  % sigproc.bib is the name of the Bibliography in this case
%\bibliography{../../../sale/big}
\bibliography{../big}
%\bibliographystyle{plain}



\end{document}