\section{Manual annotation}
There are two methods of manually annotating data in GATE. Usually,
it is quicker to annotate by using the system to produce the majority
of annotations automatically, and then manually editing the
annotations produced.

\subsection{Simple method of adding annotations}
The simple method is when only the annotation type is needed, with no
features. Annotations can be added simply by selecting the text and
clicking on the annotation type that is displayed in the list of
types.

\subsection{Complex method of adding annotations}
The more complex way of annotating documents is via the annotation
editor, which is invoked via the menu displayed on right mouse click
(after selecting the text to be annotated). This method is used in two
cases: (i) when no annotations of the desired type already exist. The
first annotation of this type needs to be created via the
editor. Subsequent annotations can then be created either via the
simple point-and-click method described above.  (ii) when we want to
annotate with more detailed information by specifying annotation
features. For example, Date annotations in GATE can be several kinds
(time, date, and date-time).

\subsection{Modifying and removing annotations}
To modify or remove an annotation, select the annotation from the annotations
table, right click and select ``Edit'' or ``Delete selected
annotations''.
To delete an entire annotation type, select the annotation type from
the annotation set on the right hand pane and use the keyboard Delete
key. 
To delete an entire annotation set, select the annotation set in the
right hand pane and use the keyboard Delete key. Note that the Default
annotation set can be cleared but not removed.


\section{Evaluation}
There are two main tools for evaluation: 
\begin{itemize}
\item Annotation Diff, which enables comparison of key and system
annotations on a document;
\item Corpus Benchmark Tool, which enables comparison of annotations
on a whole corpus and over time. 
\end{itemize}
Both tools give figures for precision, recall, F-measure and false positives. 

\subsection{Annotation Diff}
Run the Annotation Diff tool by selecting it from the Tools
menu. Select the key and response documents to be used (note that both
must be open in GATE), the annotation sets to be used for each, and the
annotation type to be evaluated. 

The tool automatically intersects all the annotation types from the
selected key annotation set with all types from the response set.  On
a separate note, you can perform a diff on the same document, between
two different annotation sets. One annotation set could contain the
key type and another could contain the response one.

After the type has been selected, the user is required to decide how
the features will be compared. Features are compared by by analyzing
if features from the key set are contained in the response set. Both
feature name and value must be identical to obtain a match.

There are three basic options to select: 
\begin{enumerate}
\item Take all the features from the key set into consideration;
\item Take only the user selected ones;
\item Ignore all features.
\end{enumerate}

If false positives are to be measured, select the annotation type (and
relevant annotation set) to be used as the denominator (normally,
Token or Sentence). The weight for the F-Measure can also be changed -
by default it is set to 0.5 (i.e. to give precision and recall equal
weight). Finally, click on "Evaluate" to display the results. Note
that the window may need to be resized manually, by dragging the
window edges or internal bars as appropriate).

In the main window, the key and response annotations will be
displayed. They can be sorted by any category by clicking on the
relevant column header. The key and response annotations will be
aligned if their indices are identical, and are color coded according
to the legend displayed.


\subsection{Corpus Benchmark tool}

To use the corpus benchmark tool, first make sure the properties of
the tool have been set correctly (see below). Then select ``Corpus
Benchmark Tool'' from the Options menu. There are 3 ways in which it
can be run:

\begin{itemize}
\item \textbf{Default mode} compares the stored
processed set with the current processed set and the human-annotated
set. This will give information about how well the system is doing
compared with a previous version.

\item \textbf{Human marked against stored processing results}
compares the stored processed set with the human-annotated set. 

\item \textbf{Human marked against current processing results}
compares the current processed set with the human-annotated set.
\end{itemize}

Once the mode has been selected, choose the directory where the corpus
is to be found. The corpus must have a directory structure consisting
of ``clean'' and ``marked'' subdirectories. The clean directory should
contain the raw texts; the marked directory shuold contain the
human-annotated texts. Finally, select the application to be run on
the corpus (for ``default'' and ``human v current'' modes).

If the tool is to be used in Default or Current mode, the corpus must
first be processed with the current set of resources. This is done by
selecting ``Store corpus for future evaluation'' from the Corpus
Benchmark Tool. Select the corpus to be processed (from the top of
the subdirectory structure, i.e. the directory containing the marked
and stored subdirectories). If a ``processed'' subdirectory exists, the
results will be placed there; if not, one will be created.

Once the corpus has been processed, the tool can be run in Default or
Current mode. The resulting HTML file will be output in the main GATE
messages window. This can then be pasted into a text editor and viewed
in an internet browser for easier viewing.

The tool can be used either in verbose or non-verbose mode, by
selecting the verbose option from the menu. In verbose mode, any score
below the user's pre-defined threshold (stored in
corpus\_tool.properties file) will show the relevant annotations for
that entity type, thereby enabling the user to see where problems are
occurring. 

\subsection{How to define the properties of the benchmark tool}
The properties of the benchmark tool are defined in the file
corpus\_tool.properties, which should be located in the directory from
which Gate is run (usually gate/build or gate/bin).

The following properties should be set:
\begin{itemize}
\item the threshold for the verbose mode (by default this is set to 0.5);
\item the name of the annotation set containing the human-marked
annotations (annotSetName);
\item the name of the annotation set containing the system-generated
annotations (outputSetName);
\item the annotation types to be considered (annotTypes).
\end{itemize}



