\documentclass[10pt,a4paper]{article}
\usepackage{lrec2006}
\usepackage{url,moreverb,hhline,graphicx,lmodern}
\usepackage[T1]{fontenc}
\usepackage[british]{babel}
\newcommand{\sluga}{\hspace{0.5em}}
\newcommand{\slugb}{\hspace{1.0em}}
\newcommand{\slugc}{\hspace{1.5em}}
%%%%%
% Message-ID: <160220101402163735%gregorio@math.unipd.it>
\renewcommand{\thesection}{\arabic{section}}
\renewcommand{\thesubsection}{\thesection.\arabic{subsection}}
\renewcommand{\thesubsubsection}{\thesubsection.\arabic{subsubsection}}
\makeatletter
\renewcommand*{\@seccntformat}[1]{%
  \csname the#1\endcsname
  \ifcsname adddot@#1\endcsname.\fi\quad
}
\let\adddot@section=T
\let\adddot@subsection=T
\let\adddot@subsubsection=T
\makeatother
%%%%%
\title{Effective Development with GATE and Reusable Code for Semantically Analysing Heterogeneous
  Documents} \name{Adam Funk, Kalina Bontcheva}
\address{Department of Computer Science\\
  University of Sheffield\\
  Regent Court, Sheffield, S1~4DP, UK\\
  a.funk@dcs.shef.ac.uk, k.bontcheva@dcs.shef.ac.uk\\}
%
\abstract{We present a practical problem that involves the analysis of a large dataset of
  heterogeneous documents obtained by crawling the web for information related to web services.
  This analysis includes information extraction from natural-language (HTML and PDF) and
  machine-readable (WSDL) documents using NLP and other techniques, classifying documents as well as
  services (defined by sets of documents), and exporting the results as RDF for use in the back-end
  of a portal that uses Web 2.0 and Semantic Web technology.  Triples representing manual
  annotations made on the portal are also exported back to our application to evaluate parts of our
  analysis and for use as training data for machine learning (ML).  This application was implemented
  in the GATE framework and successfully incorporated into an integrated project, and included a
  number of components shared with our group's other projects.}
%
\begin{document}
\maketitleabstract
%%%%%
\section{Introduction}
\label{sec:intro}
%%%%%
The Service-Finder project addresses the problem of web service discovery for a wide audience
through a portal\footnote{\url{http://demo.service-finder.eu/}} which presents automatic semantic
descriptions of a wide range of publicly available web services and also enables service consumers
(not just providers) to enrich them according to Web 2.0 principles (manual annotation according to
the project ontology, tagging, and wiki-like editing of free text fields).  In this project, the
Service Crawler (SC) carries out focused crawling for web services and archives WSDL files and
related HTML files, then passes monthly batches of these data to the Automatic Annotator (AA), which
analyses them to produce semantic annotations to the Conceptual Indexer and Matchmaker (CIM), the
semantic repository and back end for the web portal.  Additional components include the portal
itself and the clustering engine that provides recommendations. \cite{DellaValle08}

Here we present the implementation of the Automatic Annotator using the versatile
GATE\footnote{\url{http://gate.ac.uk/}} \cite{Cun02b} framework for NLP and related applications.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Input and output}
\label{sec:input}
%%%%%
The SC (Service Crawler) component delivers to the AA a monthly batch of data, consisting of a
number of compressed Heritrix \cite{heritrix-manual} Internet Archive files (up to 100~MB each),
along with an index of all the documents in the batch.  The documents for each service include one
or more WSDL files, an abstract\footnote{The \emph{abstract} is an HTML file that the SC compiles
  from various elements' and attributes' strings (the service name, documentation, operation names,
  input and output parameters, etc.) in the best WSDL file for the service.}, and zero or more HTML
and PDF files (especially those with contact details, links to WSDL files, pricing information,
terms and conditions, and other useful information).

Figure~\ref{fig:sc-index} shows a sample extract from the index, which lists every document in the
crawler output, along with the archive file number and offset where it can be found and type codes
(e.g., \texttt{w} for WSDL or \texttt{a} for abstract).  Each stanza is headed by a service
URI\footnote{The crawler generates URIs for instances of the \emph{Service} and \emph{Provider}
  classes.  The service URI always consists of the provider URI followed by one path-element, so the
  provider URI can be easily obtained from the service URI.}, which is also used in the Heritrix
archives as the URL of the abstract.  The same service URI may occur several times in this index
with different documents listed below it.
%%
\input{figures/sc-index}


This collection of files is downloaded onto one of our servers for processing, as described in the
rest of the paper.  The results are exported as RDF-XML to the CIM.  Table~\ref{tab:input-output}
summarizes the input and output of a typical monthly batch of data.
%%
\input{figures/input-output}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Annotation tasks}
\label{sec:tasks}
%%%%%
The Automatic Annotator's principal tasks are as follows:
\begin{itemize}
\item analyse WSDL files to produce \emph{Endpoint}, \emph{Interface}, and \emph{Operation}
  instances as well as properties associating them with each other and with the relevant
  \emph{Service} instances;
\item classify documents by type (e.g., documentation, pricing, contact details) and rate them as
  low-, medium-, or high-interest;
\item carry out information extraction to identify providers' addresses, phone numbers, e-mail
  addresses, etc.;
\item carry out information extraction over services to identify service level agreements, free
  trials, etc.;
\item categorize each service in one or more of the 59 subclasses of \emph{ServiceCategory}.
\end{itemize}
Some tasks require information to be amalgamated across various sets of documents, and all the
output is expressed as RDF-XML according to the project ontology.  Figure~\ref{fig:gate} highlights
several types of keywords and annotated pieces of information in GATE's GUI.
%%
\begin{figure*}[tb]
\begin{center}
\includegraphics[width=0.95\textwidth]{figures/gate-cropped}
\end{center}
\caption{Annotated document in GATE}
\label{fig:gate}
\end{figure*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Implementation}
\label{sec:impl}
%%%%%
We implemented the Automatic Annotator tools using GATE, a versatile, extensible library and
framework for NLP and text processing.  We used the \emph{GATE Developer} GUI environment to develop
some of the pipelines (using the ANNIE information extraction pipeline as a starting point), GATE's
gazetteer and JAPE tools for rapid development of processing resources (PRs) to mark keywords and
phrases with weights according to context, and PRs for more complicated functions functions that
could not be easily coded in JAPE, along with control programs, both written in Java using the
\emph{GATE Embedded} library. \cite{Cun02b}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Preprocessing}
\label{sec:preprocessing}
%%%%%
First of all, we recognized the need to split the large input dataset into manageable chunks and
took advantage of the independent nature of the data about each provider.  We developed a
preprocessor in Java that can be run from the command-line or a shell script in GNU
screen\footnote{\url{http://www.gnu.org/software/screen/}} on a server.  This preprocessor reads the
index file into memory, creates several (typically 30) serially numbered GATE Serial
DataStores\footnote{A Serial DataStore provides disk-based persistence for documents and corpora
  using Java serialization.}, then iterates through all the documents in the input archive files.
For each document, it checks the HTTP status code and discards the document if the code is 3xx, 4xx,
or 5xx; otherwise, it uses the index to identify the document type (WSDL, abstract, HTML, PDF) and
the service and provider URIs.  It then calculates the MD5 hash \cite{RFC1321} of the document and
checks the list of already hashed documents; in case of a match, it reloads the existing serialized
GATE document, merges the additional document URL and service URI into it, re-saves it, and goes on
to the next document.  Table~\ref{tab:preprocessing} shows the effects of these suppression and
de-duplication steps, and Table~\ref{tab:aa-times} shows the striking ``before and after'' effects
on performance of making more effort in the preprocessor to eliminate unnecessary documents from
analysis (on a Xeon X3220 server with Java's \texttt{-Xmx 4000m} setting).  The preprocessing time
increased sixfold but the total time decreased by 53\%.
%%
\input{figures/preprocessing}


For each valid, non-duplicate document, the preprocessor instantiates a GATE document in a
mark-up-aware manner, with the plain-text content, the HTML mark-up or XML tags, and the metadata
all stored in the appropriate parts of GATE's document model (the document content, \emph{Original
  markups} annotation set, and document feature map, respectively).  (GATE uses a modified form of
the TIPSTER and Atlas formats \cite{Gri96b,Bir99} as stand-off mark-up, as shown in
Figure~\ref{fig:gate}.)  (WSDL documents are also analysed by software developed mainly by another
project partner, seekda\footnote{\url{http://seekda.com/}}, integrated so that it stores its results
as an RDF-XML document feature.)  It adds this document to the provider's corpus (which it creates
when it first encounters that provider).  The corpora are allocated in a loop over the datastores so
that the latter end up with roughly the same sizes.  The provider--corpus and document--MD5 mappings
are kept in memory since they are used so frequently.  Finally, the preprocessor closes all the
corpora (synchronizing them on disk), closes the datastores, and writes several index files that
indicate which providers and services are in which corpora and datastores.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Analysis}
\label{sec:analysis}
%%%%%
We then run a shell script that carries out the main analysis tasks and generates the consolidated
RDF-XML file separately over each datastore.

As mentioned earlier, we took ANNIE as the starting point for the information extraction tasks, but
modified it so that most of its PRs run only on HTML and PDF documents.


Each datastore produced by the preprocessor is processed through a Java tool that includes
ANNIE\footnote{ANNIE is the information extraction system supplied with GATE; it includes standard
  NLP tools for English (such as a tokenizer and POS tagger) and gazetteers and rule-based tools for
  named entity recognition.}  with additional gazetteers and rules developed within Service-Finder
to identify relevant terms in the web service domain and annotate interesting sections and
documents, consolidate the information from various documents for each provider, merge in the
RDF-XML snippets generated by the preprocessor and attached to the corpora and documents, and
produce one large RDF-XML file for each block (datastore).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Overview}
\label{sec:analysis:overview}
%%%%%
The analysis pipeline consists of the following series of processing resources (PRs), as illustrated
in Figure~\ref{fig:ie-flowchart}.
%%
\begin{figure}[tb]
\begin{center}
\includegraphics[width=0.95\columnwidth]{figures/ie-flowchart}
\end{center}
\caption{Overview of the IE pipeline}
\label{fig:ie-flowchart}
\end{figure}
%%
\begin{enumerate}
\item Standard ANNIE components tokenize and sentence-split the HTML and PDF documents (creating
  \emph{Token} and \emph{Sentence} annotations on the document).  Abstracts and WSDLs are processed
  with a \emph{source-code tokenizer} (developed in the TAO
  project\footnote{\url{http://www.tao-project.eu/}}), a version of the ANNIE tokenizer with the
  rule files modified to split camel-cased strings (e.g., \texttt{getUnsplicedSequence} $\to$
  \texttt{get Unspliced Sequence}) as well as tokens separated by whitespace.
\item The ANNIE gazetteers and NER (named-entity recognition) module (consisting of JAPE
  transducers) identify and annotate a range of entities such as \emph{Date}, \emph{Person},
  \emph{Organization}, and \emph{Address}.
  %%
\item Gazetteers developed for this application mark keywords relating to web services, such as
  those used to indicate free trials, terms and conditions, pricing, and categories of services.
  Figure~\ref{fig:gazetteer-terms} shows keywords associated with the \emph{TermsAndConditionsPage}
  document type.
  %%
  \input{figures/gazetteer-terms}
  %%
\item A series of custom JAPE transducers compare the annotations produced by ANNIE and the custom
  gazetteers with the HTML mark-up in order to evaluate their role and set a \emph{multiplier}
  feature accordingly, which is used by the voting processor in step~\ref{step:voting}.  (For
  example, keywords and named-entity annotations in \texttt{title} or \texttt{h1} elements are more
  important than elsewhere in the document, and \texttt{p} elements that begin with the keyword
  \emph{Description} or \emph{Price} are more likely to relate to the description of the service or
  its pricing.  Such annotations are assigned a \emph{multiplier} feature $1{\leq}m$, typically
  $m<6$.)  Other JAPE rules try to identify company details such as country of origin.
  \label{step:multiplier}

  These rules also have access to document features containing metadata such as the service,
  provider, and document URIs, which they can copy into features on the annotations they create.
  They also adjust an \emph{interesting} feature\footnote{$0.0{\leq}i{\leq}3.0$, interpreted as low
    ($0.0{\leq}i<1.0$), medium ($1.0{\leq}i<2.0$), or high ($2.0{\leq}i{\leq}3.0$).  All documents
    start with $0.0$.}\label{step:interesting}\label{step:last-before-ml}.
  %%
\item\label{step:ml} An instance of the GATE machine learning PR \cite{Yaoyong09a} aims to label
  each document with a class from the service category ontology.  Section~\ref{sec:categorization}
  describes the integration of this component in more detail.
  %% 
\item\label{step:voting} ``Voting'' PRs (extensions of \emph{AbstractLanguageAnalyser} from the
  \emph{GATE Embedded} library) compare the weighted frequency of significant keywords to assign
  certain ontology classes and properties as follows.
  \begin{itemize}
  \item For each WSDL document and interesting (see step~\ref{step:interesting} above) textual
    (i.e., HTML or PDF) document, create a potential instance of the general class \emph{Document}
    or one of its subclasses (\emph{DocumentationPage}, \emph{TermsAndConditionsPage}, etc.)  and
    assert the properties \emph{hasSize}, \emph{hasTitle}, and \emph{retrievedAt}; also create a
    \emph{DocumentAnnotation} associating the document with a service or provider.  (Some document
    types, such as \emph{ContactDetailsPage}, are associated with providers; others are associated
    with services.)
  \item For each service, assertions of the properties \emph{supportsAuthenticationModel},
    \emph{hasServiceLevelAgreement}, \emph{allowsFreeTrials}, etc.; these properties are not
    asserted if no votes (keywords) are found.
  \item For each service, the two best categories for \emph{CategoryAnnotation} (or just one
    category if all the votes were for the same one); the categories assigned by machine-learning
    outweigh those generated by keywords and rules (as \S\ref{sec:categorization} explains in
    detail).
  \item For each provider, one or two best values for \emph{hasHomepage}, \emph{fromCountry},
    \emph{hasEmail}, \emph{hasAddress}, and \emph{hasTelephone} (these properties are not asserted
    when no values are found in the documents).
  \end{itemize}

  All GATE PRs override the \texttt{execute} method, which is called on each document in the corpus.
  Each voting PR uses additional hooks, as shown in Figure~\ref{fig:hooks}, to initialize the set of
  ``ballots'' at the beginning of the \texttt{execute} on the first document in the corpus; and to
  compute the results, generate the corresponding RDF-XML, and store it as a corpus feature at the
  end of the \texttt{execute} on the last document in the corpus.  (The RDF-XML is generated by
  matching and filling in templates, as described in \S\ref{sec:ontology}.)
  %%
  \input{figures/hooks}
  %%
\item\label{step:xml} Some important annotations on the document are translated into RDF-XML
  according to a set of templates based on the features of each annotations.  (This RDF-XML is also
  generated from templates.)
  % 
\item\label{step:final}  All the RDF-XML snippets are collected from the document and corpus
  features and consolidated into one output file for the datastore.
\end{enumerate}


The ``pipeline'' outlined above actually consists of two GATE \emph{SerialAnalyserController}
instances (conditional corpus pipelines) serialized as \texttt{gapp} files and an XML configuration
file for GATE's \emph{Batch Learning PR} \cite{Yaoyong09a}.  Like the preprocessor, the main
analysis tool is a command-line Java program that can be run in GNU screen on a server; it
instantiates the two pipelines from the \texttt{gapp} files and creates a separate pipeline for the
learning PR (which it instantiates from the configuration file); it then iterates through the
corpora in a datastore, runs the pipelines in the correct sequence (taking advantage of GATE's
serialization to save memory: only one document is loaded at a time), and finally collects together
all the RDF-XML snippets stored as document and corpus features in the datastore and consolidates
them into one RDF-XML file, which constitutes this tool's output to the CIM for that block
(datastore).


A modified version is also used in the NeOn project\footnote{\url{http://www.neon-project.org/}} for
batch processing large datastores through pipelines (both specified by command-line arguments) on a
server.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Service categorization}
\label{sec:categorization}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Although the portal's Web~2.0 features encourage users to add, correct, and otherwise improve the
category annotations of services, it is important to provide a number of reasonably good ones to
start with so users will find the portal useful and interesting and then contribute to
it---otherwise they would have to face 23~000 uncategorized services with only keyword searching.
We therefore placed strong emphasis in the AA on rapid development and then refinement of service
categorization.  In this paper, we will summarize this task and describe the integration of the
components into the pipeline above.  Our scientific results here are interesting in their own right
and are published in detail elsewhere \cite{Funk2010}.


Briefly, the task is to annotate each service with one or more of the 59 subclasses of
\emph{Category} for web services, such as \emph{Genetics}, \emph{Address Information}, and
\emph{Media Management}, arranged in a shallow tree (down to three levels below the top class) in
seven main branches, such as \emph{Business}, \emph{Consumer}, and
\emph{Science}~\cite{DellaValle08,SF-D4.2}.  


In the early stages, we had no training data but needed to generate some category annotations
quickly for the portal, so we created \emph{ad hoc} gazetteers of keywords and phrases based on the
category names, synonyms, and related words.  These were affected by the multipliers described in
step~\ref{step:multiplier} in \S\ref{sec:analysis:overview}.  The IE pipeline at this stage was as
shown in Figure~\ref{fig:ie-flowchart} but without the machine learning (ML) categorization PR
(step~\ref{step:ml} in \S\ref{sec:analysis:overview}); the voting process treated keyword or phrase
match as $m$ (the multiplier) votes for the relevant category for each service associated with the
document, and annotated each service with the two highest-scoring categories (an arbitrary limit
agreed within the project).


After the first release of the portal, we manually annotated a few hundred services with the same
portal features that users have for manually adding or correcting categories, and used these
annotations to evaluate the AA's categorization and then as training data for machine-learning.  We
trained the ML PR to classify documents, carried out evaluations with various ML parameters, and
integrated the ML PR into the pipeline by assigning a very high multiplier ($m=100$) to the ML
annotations so that in the subsequent voting PR, they will outweigh any gazetteer-based
categorizations, although the latter can still be used to make the total number of categories per
service up to two if the ML PR fails to classify any of a service's documents or provides only one
category, since it is more user-friendly for us to provide approximate categorizations than none at
all.  We have published elsewhere \cite{Funk2010} the full technical details and results of our ML
experiments.
%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Approaches to ontology development and population}
\label{sec:ontology}
%%%%%
One can develop and populate ontologies from GATE applications using the GATE Ontology API
\cite{Bon04b}, and we have used this technique in our SPRAT and SARDINE\footnote{Semantic Pattern
  Recognition and Annotation Tool; Species Annotation and Recognition and Indexing of Named
  Entities; both developed in NeOn.} applications \cite{Maynard09ck,Maynard09wop}, which use
ontology design patterns to recognize new concepts, instances, and properties, and add them to the
seed ontology (which can be empty) used to initialize the application.  The principal output of both
applications is the extended ontology produced from a corpus of documents.  As an evaluated example,
SPRAT processed 25 Wikipedia articles and produced 1058 classes, 659 subclasses, 23 instances, and
55 properties as output.

The CLOnE\footnote{Controlled Language for Ontology Editing.} and RoundTrip Oontology Authoring
software \cite{Funk07,Davis2008} developed and used in
SEKT\footnote{\url{http://www.sekt-project.com/}} and
NEPOMUK\footnote{\url{http://nepomuk.semanticdesktop.org/}}, also used the GATE Ontology API.


In the Service-Finder and MUSING tasks, however, the ontologies' class and property structures are
fixed and our applications only need to create instances and property assertions, specifically in
RDF-XML (as requested by other developers in the projects).  The volume of data generated is also
much larger and more time-consuming (as shown in Tables~\ref{tab:input-output}
and~\ref{tab:aa-times}), so we use a template-filling technique which requires relatively little
memory.


We originally developed this GATE PR for generating XML (principally RDF-XML) in the
MUSING\footnote{\url{http://www.musing.eu/}} business intelligence research project, and have used
modified and improved versions of it in Service-Finder and
CLARIN\footnote{\url{http://www.clarin.eu/}}.  (A version of it will probably be integrated into
GATE in the future, once we have settled the list of configurable features.)  The PR's configuration
file consists mainly of a series of template specifications, as Figure~\ref{fig:rdf-template} shows.
%
\input{figures/xml}
%
Each entry lists the annotation features that are required for the template to match, and the values
of those features are substituted for the variables in the template.  (The annotation can have other
features, which are ignored.)  The PR's output for a matching annotation consists of the
``filled-in'' copy of the element(s) in the \texttt{template} element.
Figures~\ref{fig:matching-mention} and~\ref{fig:rdf-snippet} show the characteristics of a matching
annotation and the resulting XML snippet, respectively.  (In addition to the \texttt{feature}
element, the entry can have a \texttt{generated} specification, which names a variable for which a
UUID string is substituted.  This is useful for generating unique \url{rdf:id} values.)


When the configuration file is loaded (during initialization of the PR), the order of the entries is
preserved, so that for each annotation, the first match is used.  It is therefore possible to use a
template that requires an annotation with three features, followed by a simpler template which
requires only two of the three (and uses a default value for the third, for example); the first
template will ``fire'' whenever all three features appear, but the PR will drop back to the second
one if the first does not match, and so on.  (Another version of this component takes a
\emph{Map<String, String>} rather than a GATE Annotation; the voting PR generates the maps from the
winning ballots and obtains the corresponding RDF-XML from this generator.)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Miscellaneous tools}
\label{sec:misc-tools}
\label{sec:archiving}
%%%%%
We developed two other separate tools for the Service-Finder AA.  The archiving tool iterates
through the documents in the GATE datastores after the analysis tool has been run (leaving its
annotations on the serialized documents) and produces Heritrix archives of the plain-text content of
``interesting'' (see step~\ref{step:interesting} in \S\ref{sec:analysis}) HTML and PDF documents and
selected strings from WSDLs and abstracts; the CIM produces Lucene indexes from these files to
support keyword searches for services on the portal.  The quantitative evaluation tool loads the
RDF-XML files into a Sesame repository, executes a series of SERQL or SPARQL queries specified in a
control file, and produces an output file containing the number of results for each query or a list
of those results (according to the specifications in the control file).

To provide a datastore suitable for training the ML classifier, we added extra features to the
preprocessor (\S\ref{sec:preprocessing}), activated by additional command-line options, so that it
reads a file of manual category annotations (exported from the CIM) as well as a set of archives
from the SC and produces a datastore containing only the documents related to the manually annotated
services, with document features representing the categories.  We also developed a pipeline for
training the classifier, which carries out steps~1 through~\ref{step:last-before-ml} in
\S\ref{sec:analysis} and treats the annotated documents as instances, and then saves the learned
model for use in the complete integrated analysis tool.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}
\label{sec:conclusion}
%%%%%
The immediate result of the development presented here was its contribution to the successful
completion of the Automatic Annotator tasks and their integration in the Service-Finder project,
which received good intermediate and final project reviews.  The relevant public deliverables
\cite{SF-D3.3,SF-D3.4} describe the AA software in much greater detail.  We evaluated the AA
software itself in two ways: IE measures (precision, recall, and $F_1$) for the especially important
and scientifically interesting service categorization task, which we present in detail elsewhere
\cite{Funk2010}; and quantitative measures of the instances and property assertions created at
various stages of development~\cite{SF-D7.5}.


The broader results included the dissemination of GATE as a tool for semantically annotating the
results of focused web crawling---in particular at a \emph{Future Internet Symposium }tutorial on
web service crawling and annotation \cite{Funk2009fis}, where we demonstrated the suitability of the
\emph{GATE Developer} IDE and \emph{GATE Embedded} library for rapid application development and
effective code re-use---and the development of useful, reusable code shared with other projects
(NeOn, MUSING, and CLARIN).
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Acknowledgements}
%%%%%
This research is partially supported by the EU Sixth Framework Program projects NeOn
(IST-2005-027595) and MUSING (FP6-027097) and the Seventh Framework Program project Service-Finder
(FP7-215876).
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% All references within the text are placed in parentheses containing the author's surname followed by
% a comma before the date of publication \cite{chomsky-73}. If the sentence already includes the
% author's name, then it is only necessary to put the date in parentheses: \newcite{aslin-49} . When
% several works are cited, those references are separated with a semicolon \cite{chomsky-73,feigl-58}:
% When the reference has more than three authors, only the name of the first author, followed by et
% al, appears \cite{fletcher-hopkins}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{lrec2006}
\bibliography{../../big,extra}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}

