% $ID$
GATE defines three different types of resources:
\begin{description}
\item[Language Resources]{: (\textbf{LRs}) entities that hold linguistic data.
}

\item[Processing Resources]{: (\textbf{PRs}) entities that process data.
}

\item[Visual Resources]{: (\textbf{VRs}) components used for building
graphical interfaces. }

\end{description}

These resources are collectively named \textbf{CREOLE}\footnote{CREOLE
stands for Collection of REusable Objects for Language Engineering} resources.

All CREOLE resources have some associated meta-data in the form of an entry
in a special XML file named {\tt creole.xml}. The most important role of
that meta-data is to specify the set of parameters that a resource
understands, which of them are required and which not, if they have default
values and what those are.

All resource types have creation-time parameters that are used during the
initialisation phase. Processing Resources also have run-time parameters
that get used during execution (see section \ref{sect:PRs} for more
details).

\textbf{Controllers} are used to define GATE applications and have the role of
controlling the execution flow (see section \ref{sect:controllers} for
more details).

%-------------------------------------
\sect{Resource Management in GATE}
\label{sect:factory}

This \sectthing\ describes how to create and delete CREOLE resources as
objects in a running Java virtual machine. This process involves using
GATE’s Factory class\footnote{Fully qualified name: \tt{gate.Factory}}, and,
in the case of LRs, may also involve using a DataStore.

CREOLE resources are Java Beans; creation of a resource object involves
using a default constructor, then setting parameters on the bean, then
calling an {\tt init()} method. The Factory takes care of all this, makes sure
that the GUI is told about what is happening (when GUI components exist at
runtime), and also takes care of restoring LRs from DataStores. \textbf{A
programmer using GATE should never call the constructor of a resource:
always use the Factory!}

Creating a resource involves providing the following information:
\begin{itemize}
\item \textbf{fully qualified class name} for the resource. This is the only
{\bf required} value. For all the rest, defaults will be used if actual values
are not provided.
\item values for the \textbf{creation time parameters}.$^\dagger\ $
\item initial values for \textbf{resource features}.$^\dagger\ $ For an
explanation on features see section \ref{subsect:features}.
\item a \textbf{name} for the new resource;
\end{itemize}

$^\dagger\ $ Parameters and features need to be provided in the form of a
GATE Feature Map which is essentially a java Map ({\tt java.util.Map})
implementation, see section \ref{subsect:features} for more details
on Feature Maps.

\begin{minipage}{\textwidth}
\eg\ Here is an example that creates a GATE document:

\HBar\
\begin{verbatim}
URL u = new URL("http://gate.ac.uk/");
FeatureMap params = Factory.newFeatureMap();
params.put("sourceUrl", u);
FeatureMap features = Factory.newFeatureMap();

Document doc = (Document)
  Factory.createResource("gate.corpora.DocumentImpl",
                         params, features, "GATE Homepage");
\end{verbatim}
\HBar\
\end{minipage}

Apart from {\tt createResource()} methods with different signatures, {\tt
Factory} also provides some shortcuts for common operations:

\begin{description}
%
\item[{\tt newFeatureMap()}] creates a new Feature Map (as used in the
example above).
%
\item[{\tt newDocument(String content)}] creates a new GATE Document starting
from a String value that will be used to generate the document content.
%
\item{{\tt newDocument(URL sourceUrl)}} creates a new GATE Document
using the text pointed by an URL  to generate the document content.
%
\item[{\tt newDocument(URL sourceUrl, String encoding)}] same as
above but allows the specification of an encoding to be used while
downloading the document content.
%
\item[{\tt newCorpus(String name)}] creates a new GATE Corpus with a specified
name.
%
\end{description}

GATE maintains various data structures that allow the retrieval of loaded
resources. When a resource is no longer required, it needs to be removed from
those structures in order to remove all references to it, thus making it a
candidate for garbage collection. This is achieved using the \linebreak\ {\tt
deleteResource(Resource res)} method on {\tt Factory}.

\ans\ Simply removing all references to a resource from the user code will
{\bf NOT} be enough to make the resource collect-able. Not calling \linebreak\ 
{\tt Factory.deleteResource()} {\bf will} lead to memory leaks!

%-------------------------------------
\sect{Language Resources} \label{sect:lr}

This \sectthing\ describes the
implementation of documents and corpora in GATE.

\subsect{GATE Documents}

Documents are modelled as content plus annotations (see section
\ref{subsect:annotations}) plus features (see section
\ref{subsect:features}).

The content of a document can be any implementation of the \linebreak\ {\tt
gate.DocumentContent} interface; the features are \verb!<!attribute,
value\verb!>! pairs stored a Feature Map. Attributes are String values while
the values can be any Java object.

The annotations are grouped in sets (see section \ref{subsect:annSets}). A
document has a default (anonymous) annotations set and any number of named
annotations sets.

Documents are defined by the {\tt gate.Document} interface and the provided
implementations are:
\begin{description}
\item[{\tt gate.corpora.DocumentImpl}]: transient document. Can be stored
persistently through Java serialisation.
%
\item[{\tt gate.corpora.DatabaseDocumentImpl}]: database persistent
documents. It can be used in conjunction with database
datastores.\footnote{Currently implementations for ORACLE and PostgreSQL are
provided.}
%
\end{description}

Main Document functions are presented in table \ref{table:document}.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{|p{.5\textwidth}|p{.5\textwidth}|}
\hline
\multicolumn{2}{|c|}{\textbf{Content Manipulation}}\\
\hline
\hline
\textbf{Method} & \textbf{Purpose}\\
\hline
{\tt DocumentContent {\bf getContent}()} & Gets the Document content. \\
\hline
{\tt void {\bf edit}(Long start, Long end, DocumentContent replacement)} &
Modifies the Document content.\\
\hline
{\tt void {\bf setContent}(DocumentContent newContent)} & Replaces the entire
content.\\
\hline
\multicolumn{2}{|c|}{\textbf{Annotations Manipulation}}\\
\hline
\textbf{Method} & \textbf{Purpose}\\
\hline
{\tt public AnnotationSet {\bf getAnnotations}()} & Returns the default annotation
set.\\
\hline
{\tt public AnnotationSet {\bf getAnnotations}(String name)} & Returns a {\em
named} annotation set.\\
\hline
{\tt public Map {\bf getNamedAnnotationSets}()} & Returns {\em all} the named
annotation sets.\\
\hline
{\tt void {\bf removeAnnotationSet}(String name)} & Removes a named
annotation set.\\
\hline
\multicolumn{2}{|c|}{\textbf{Input Output}}\\
\hline
{\tt String {\bf toXml}()} & Serialises the Document in XML format.\\
\hline
{\tt String {\bf toXml}(Set aSourceAnnotationSet, boolean includeFeatures)}
& Generates XML from a set of annotations only, trying to preserve the
original format of the file used to create the document.\\ 
\hline
\end{tabular}
\caption{{\tt gate.Document} methods.}
\label{table:document}
\end{center}
\end{table}


\subsect{Feature Maps}
\label{subsect:features}
All CREOLE resources as well as the {\em Controllers} and the annotations
can have attached meta-data in the form of {\em Feature Maps}.

A Feature Map is a Java Map (i.e. it implements the {\tt java.util.Map}
interface) and holds \verb!<!attribute-name, attribute-value\verb!>! pairs.
The attribute names are Strings while the values can be any Java Objects.

The use of non-{\em Serialisable} objects as values is strongly discouraged.

Feature Maps are created using the {\tt gate.Factory.newFeatureMap()}
method.

The actual implementation for FeatureMaps is provided by the \linebreak\ {\tt
gate.util.SimpleFeatureMapImpl} class.

Objects that have features in GATE implement the {\tt
gate.util.FeatureBearer} interface which has only the two accessor methods
for the object features: {\tt FeatureMap getFeatures()} and {\tt void
setFeatures(FeatureMap features)}.

\begin{minipage}{\textwidth}
\eg\ {\it How to get a particular feature from an object?}

\HBar\
\begin{verbatim}
Object obj;
String featureName = "length";
if(obj instanceof FeatureBearer){
  FeatureMap features = ((FeatureBearer)obj).getFeatures();
  Object value = (features == null) ? null :
                                      features.get(featureName);
}
\end{verbatim}
\HBar\
\end{minipage}

\subsect{Annotation Sets}
\label{subsect:annSets}

A GATE document can have one or more annotation layers --- an anonymous one,
(also called {\em default}), and as many {\em named} ones as necessary.

An annotation layer is organised as a \emph{Directed Acyclic Graph (DAG)} on
which the nodes are particular locations ---\emph{anchors}--- in the document content
and the arcs are made out of annotations reaching from the location indicated
by the start node to the one pointed by the end node (see Figure
\ref{fig:ann-graph} for an illustration). Because of the \emph{graph} metaphor,
the annotation layers are also called \emph{annotation graphs}.
In terms of Java objects, the annotation layers are represented using
the \emph{Set} paradigm as defined by the collections library and they are hence
named \emph{annotation sets}. The terms of annotation \emph{layer}, \emph{graph}
and \emph{set} are interchangeable and refer to the same concept when used in
this \thing{}.

\begin{figure}[htbp]
\begin{center}
\includegraphics[scale=0.5]{annotationGraph}
\caption{The Annotation Graph model.}
\label{fig:ann-graph}
\end{center}
\end{figure}

An annotation set holds a number of annotations and maintains a series of
indices in order to provide fast access to the contained annotations.

The GATE Annotation Sets are defined by the {\tt gate.AnnotationSet}
interface and the current implantations are:

\begin{description}
%
\item[{\tt gate.annotation.AnnotationSetImpl}] annotation set implementation
used by transient documents.
%
\item[{\tt gate.annotation.DatabaseAnnotationSetImpl}] annotation set
implementation used by persistent documents.
%
\end{description}

The annotation sets are created by the document as required. The first time
a particular annotation set is requested from a document it will be
transparently created if it doesn't exist.

Tables \ref{table:annSet1} and \ref{table:annSet2} list the most used
Annotation Set functions.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{|p{.5\textwidth}|p{.5\textwidth}|}
\hline
\multicolumn{2}{|c|}{\textbf{Annotations Manipulation}}\\
\hline
\hline
\textbf{Method} & \textbf{Purpose}\\
\hline
{\tt Integer {\bf add}(Long start, Long end, String type, FeatureMap features)}
& Creates a new annotation between two offsets, adds it to this set and returns its id.\\
\hline
{\tt Integer {\bf add}(Node start, Node end, String type, FeatureMap
features)} & Creates a new annotation between two nodes, adds it to this
set and returns its id.\\
\hline
{\tt boolean {\bf remove}(Object o)} & Removes an annotation from this
set.\\
\hline

\multicolumn{2}{|c|}{\textbf{Nodes}}\\
\hline
\textbf{Method} & \textbf{Purpose}\\
\hline
{\tt Node {\bf firstNode}()} & Gets the node with the smallest offset.\\
\hline
{\tt Node {\bf lastNode}()} & Gets the node with the largest offset.\\
\hline
{\tt Node {\bf nextNode}(Node node)} & Get the first node that is relevant
for this annotation set and which has the offset larger than the one of the
node provided.\\
\hline
\multicolumn{2}{|c|}{\textbf{Set implementation}}\\
\hline
{\tt Iterator {\bf iterator}() } & \\
\hline
{\tt int {\bf size}()} & \\
\hline
\end{tabular}
\caption{{\tt gate.AnnotationSet} methods (general purpose).}
\label{table:annSet1}
\end{center}
\end{table}


\begin{table}[htbp]
\begin{center}
\begin{tabular}{|p{.5\textwidth}|p{.5\textwidth}|}
\hline
\multicolumn{2}{|c|}{\textbf{Searching}}\\
\hline
{\tt AnnotationSet {\bf get}(Long offset)} & Select annotations by offset.
This returns the set of annotations whose start node is the least such that
it is less than or equal to offset. If a positional index doesn't exist it
is created. If there are no nodes at or beyond the offset parameter then it
will return null.\\
\hline
{\tt AnnotationSet {\bf get}(Long startOffset, Long endOffset)} &
Select annotations by offset. This returns the set of annotations that
overlap totally or partially with the interval defined by the two provided
offsets. The result will include all the annotations that either:
\begin{itemize}
\item start before the start offset and end strictly after it
\item start at a position between the start and the end offsets
\end{itemize}\\
\hline
{\tt AnnotationSet {\bf get}(String type)} & Returns all annotations of the
specified type.\\
\hline
{\tt AnnotationSet {\bf get}(Set types)} & Returns all annotations of the
specified types.\\
\hline
{\tt AnnotationSet {\bf get}(String type, FeatureMap constraints)} & Selects
annotations by type and features.\\
\hline
{\tt Set {\bf getAllTypes}()} & Gets a set of java.lang.String objects
representing all the annotation types present in this annotation set.\\
\hline
\end{tabular}
\caption{{\tt gate.AnnotationSet} methods (searching).}
\label{table:annSet2}
\end{center}
\end{table}

\begin{minipage}{\textwidth}
\eg\ {\it How to iterate from left to right over all annotations of a given
type?}

\HBar\
\begin{verbatim}
AnnotationSet annSet = ...;
String type = "Person";
//Get all person annotations
AnnotationSet persSet = annSet.get(type);
//Sort the annotations
List persList = new ArrayList(persSet);
Collections.sort(persList, new gate.util.OffsetComparator());
//Iterate
Iterator persIter = persList.iterator();
while(persIter.hasNext()){
...
}
\end{verbatim}
\HBar\
\end{minipage}

%--------------------------
\subsect{Annotations}
\label{subsect:annotations}

An \textbf{annotation}, is a form of meta-data attached to a particular
section of document content. The connection between the annotation and the
content it refers to is made by means of two pointers that represent the
start and end locations of the covered content. An annotation must also have
a type (or a name) which is used to create classes of similar annotations,
usually linked together by their semantics.

An Annotation is defined by:
\begin{description}
\item[start node] a location in the document content defined by an offset.
\item[end  node] a location in the document content defined by an offset.
\item[type] a String value.
\item[features] (see \sectthing\ \ref{subsect:features}).
\item[ID] an Integer value. All annotations IDs are unique inside an
annotation set.
\end{description}

In GATE, annotations are defined by the {\tt gate.Annotation} interface and
implemented by the {\tt gate.annotation.AnnotationImpl} class.
Annotations exist only as members of annotation sets (see
\sectthing\ \ref{subsect:annSets}) and they should not be directly created by
means of a constructor. Their creation should always be delegated to the
containing annotation set.

\subsect{GATE Corpora}
\label{subsect:corpora}

A corpus in GATE is a Java List (i.e. an implementation of {\tt
java.util.List}) of documents. GATE corpora are defined by the {\tt
gate.Corpus} interface and the following implementations are available:
\begin{description}
\item[{\tt gate.corpora.CorpusImpl}] used for transient corpora.
\item[{\tt gate.corpora.SerialCorpusImpl}] used for persistent corpora that
are stored in a serial datastore (i.e. as a directory in a file system).
\item[{\tt gate.corpora.DatabaseCorpusImpl}] used for persistent corpora
stored in a database datastore.
\end{description}

Apart from implementation for the standard List methods, a Corpus also
implements the methods in table \ref{table:corpus}.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{|p{.5\textwidth}|p{.5\textwidth}|}
\hline
\textbf{Method} & \textbf{Purpose}\\
\hline
{\tt String {\bf getDocumentName}(int index)} & Gets the name of a document
in this corpus.\\
\hline
{\tt List {\bf getDocumentNames}()} & Gets the names of all the documents
in this corpus.\\
\hline
{\tt void {\bf populate}(URL directory, FileFilter filter, String encoding,
boolean recurseDirectories)} & Fills this corpus with documents created on
the fly from selected files in a directory. Uses a {\tt FileFilter} to
select which files will be used and which will be ignored. A simple file
filter based on extensions is provided in the Gate distribution ({\tt
gate.util.ExtensionFileFilter}).\\

\hline
\end{tabular}
\caption{{\tt gate.Corpus} methods.}
\label{table:corpus}
\end{center}
\end{table}

\begin{minipage}{\textwidth}
\eg\ {\it How to create a corpus from all XML files in a directory?}

\HBar\
\begin{verbatim}
Corpus corpus = Factory.newCorpus("My XML Files");
File directory = ...;
java.io.FileFilter filter = new gate.util.ExtensionFileFilter();
filter.addExtension("xml");
URL url = directory.toURL();
corpus.populate(url, filter, null, false);
\end{verbatim}
\HBar\
\end{minipage}


\sect{Ontologies}
Starting from GATE version {\tt 3.1}, support for ontologies has been added.
Ontologies are nominally Language Resources but are quite different from
documents and corpora and are detailed in this section dedicated to them.

Classes related to ontologies are to be found in the {\tt gate.creole.ontology}
package and its sub-packages. The top level package defines an abstract API for
working with ontologies while the sub-packages contain concrete implementations.
The GATE API for ontologies does not provide full support for all operations
that are posible using different ontology models (like RDF-S, OWL or DAML); it
only tries to provide support for all the operations that all these formalisms
have in common. This includes hierarchies of classes, instances (or individuals)
and properties. The terminology used in the naming of the classes and the
operations is sligthly biased toward RDF-S and OWL.

The entry point to the ontology API is the {\tt gate.creole.ontology.Ontology}
interface which is the base interface for all concrete implementations. It
provides methods for accessing the class hierarchy, listing the instances and
the properties.

\sect{Processing Resources}
\label{sect:PRs}

Processing Resources ({\bf PRs}) represent entities that are primarily
algorithmic, such as parsers, generators or ngram modellers.

They are created using the GATE Factory in manner similar the Language
Resources. Besides the creation-time parameters they also have a set of
run-time parameters that are set by the system just before executing them.

Analysers are a particular type of processing resources in the sense that
they always have a {\bf {\tt document}} and a {\bf {\tt corpus}} among their
run-time parameters.

The most used methods for Processing Resources are presented in table
\ref{table:pr}


\begin{table}[htbp]
\begin{center}
\begin{tabular}{|p{.5\textwidth}|p{.5\textwidth}|}
\hline
\textbf{Method} & \textbf{Purpose}\\
\hline
{\tt void {\bf setParameterValue}(String paramaterName, Object
parameterValue)} & Sets the value for a specified parameter. {\small method
inherited from {\tt gate.Resource}}\\
\hline
{\tt void {\bf setParameterValues}(FeatureMap parameters)} & Sets the
values for more parameters in one step. {\small method inherited from {\tt
gate.Resource}}\\
\hline
{\tt Object {\bf getParameterValue}(String paramaterName)} & Gets the value
of a named parameter of this resource. {\small method inherited from {\tt
gate.Resource}}\\
\hline
{\tt Resource {\bf init}()} & Initialise this resource, and return it.
{\small method inherited from {\tt gate.Resource}}\\
\hline
{\tt void {\bf reInit}()} & Reinitialises the processing resource. After
calling this method the resource should be in the state it is after calling
init. If the resource depends on external resources (such as rules files)
then the resource will re-read those resources. If the data used to create
the resource has changed since the resource has been created then the
resource will change too after calling reInit().\\
\hline
{\tt void {\bf execute}()} & Starts the execution of this Processing
Resource.\\
\hline
{\tt void {\bf interrupt}()} & Notifies this PR that it should stop its
execution as soon as possible.\\
\hline
{\tt boolean {\bf isInterrupted}()} & Checks whether this PR has been
interrupted since the last time its Executable.execute() method was
called.\\
\hline
\end{tabular}
\caption{{\tt gate.ProcessingResource} methods.}
\label{table:pr}
\end{center}
\end{table}

\sect{Controllers}
\label{sect:controllers}

Controllers are used to create GATE Applications. A Controller handles a set of
Processing Resources and can execute them following a particular strategy.
GATE provides a series of serial controllers (i.e. controllers that run
their PRs in sequence):

\begin{description}
\item[{\bf {\tt gate.creole.SerialController}:}] a serial controller that takes any
kind of PRs.
\item[{\bf {\tt gate.creole.SerialAnalyserController}:}] a serial controller that
only accepts Language Analysers as member PRs.
\item[{\bf {\tt gate.creole.ConditionalSerialController}:}] a serial controller
that accepts all types of PRs and that allows the inclusion or exclusion of
member PRs from the execution chain according to certain run-time conditions
(currently features on the document being processed are used).
\item[{\bf {\tt gate.creole.ConditionalSerialAnalyserController}:}] a serial
controller that only accepts Language Analysers and that allows the conditional
run of member PRs.
\end{description}

\begin{minipage}{\textwidth}
\eg\ {\it The following example shows how to create an ANNIE application
and how to run it over a corpus}

\HBar\
\begin{verbatim}
// load the ANNIE plugin
Gate.getCreoleRegister().registerDirectories(new File(
 Gate.getPluginsHome(), "ANNIE").toURI().toURL());

// create a serial analyser controller to run ANNIE with
SerialAnalyserController annieController =
 (SerialAnalyserController) Factory.createResource(
     "gate.creole.SerialAnalyserController",
     Factory.newFeatureMap(),
     Factory.newFeatureMap(), "ANNIE");

// load each PR as defined in ANNIEConstants
for(int i = 0; i < ANNIEConstants.PR_NAMES.length; i++) {
  // use default parameters
  FeatureMap params = Factory.newFeatureMap();
  ProcessingResource pr = (ProcessingResource)
      Factory.createResource(ANNIEConstants.PR_NAMES[i],
                             params);
  // add the PR to the pipeline controller
  annieController.add(pr);
} // for each ANNIE PR

// Tell ANNIE's controller about the corpus you want to run on
Corpus corpus = ...;
annieController.setCorpus(corpus);
// Run ANNIE
annieController.execute();
\end{verbatim}
\HBar\
\end{minipage}

\sect{Persistent Applications}
\label{sect:persisApps}

GATE allows the persistent storage of applications in a format based on Java
serialisation. This is particularly useful for applications management and
distribution. A developer can save the state of an application when
he/she stops working on its design and continue developing it in a next
session. When the application reaches maturity it can be deployed to the
client site using the same method.

When an application (i.e. a {\em Controller}) is saved, GATE will actually
only save the values for the parameters used to create the Processing
Resources that are contained in the application. When the application is
reloaded, all the PRs will be re-created using the saved parameters.

Many PRs use external resources (files) to define their behaviour and, in most
cases, these files are identified using URLs. During the saving process, all
the URLs are converted relative URLs based on the location of the
application file. This way, if the resources are packaged together with the
application file, the entire application can be reliably moved to a
different location.

API access to application saving and loading is provided by means of two
static methods on the {\tt gate.util.persistence.PersistenceManager} class:

\begin{description}
\item[{\tt public static void saveObjectToFile(Object obj, File file)}:]
saves the data needed to re-create the provided GATE object to the specified
file. The Object provided can be any type of Language or Processing Resource
or a Controller. The procedures may work for other types of objects as well
(e.g. it supports most Collection types).
\item[{\tt public static Object loadObjectFromFile(File file)}:] parses the
file specified (which needs to be a file created by the above method) and
creates the necessary object(s) as specified by the data in the file.
Returns the root of the object tree.
\end{description}



\begin{minipage}{\textwidth}
\eg\ {\it The following example shows how to save and load a GATE
application}

\HBar\
\begin{verbatim}
//Where to save the application?
File file = ...;
//What to save?
Controller theApplication = ...;

//save
gate.util.persistence.PersistenceManager.
          saveObjectToFile(theApplication, file);
//delete the application
Factory.deleteResource(theApplication);
theApplication = null;

[...]
//load the application back
theApplication = gate.util.persistence.PersistenceManager.
                 loadObjectFromFile(file);
\end{verbatim}
\HBar\
\end{minipage}

