\section{Introduction}

JAPE allows you to recognise regular expressions in annotations on
documents. A JAPE grammar consists of a set of phases, each of which
consists of a set of pattern/action rules. The phases run
sequentially and constitute a cascade of finite state transducers over
annotations. The left-hand-side (LHS) of the rules consist of an
annotation pattern that may contain regular expression operators
(e.g. *, ?, +). The right-hand-side (RHS) consists of annotation
manipulation statements.  Annotations matched on the LHS of a rule may
be referred to on the RHS by means of labels that are attached to
pattern elements.


\section{Pattern matching}
A pattern is specified in terms of one or more annotations, and
optionally, the values of any or all of its features.
The following operators can be used:
\begin{itemize}
\item $\mid $  - or
\item *  - zero or more occurrences
\item ?  - zero or one occurrences 
\item +  - one or more occurrences
\end{itemize}

Note that there is no negative operator. There are, however, ways
round this problem, as will be explained later.

An operator can operate on any pattern enclosed in round brackets.
Every complete pattern to be annotated must be enclosed in round brackets and
followed by a label. A label is denoted by a preceding
semi-colon. In the example below, the label is :loc.
\begin{verbatim}
(  
 {Lookup.majorType == location}  
 )  
 :loc 
\end{verbatim}

 It is
possible to have more than one pattern (and corresponding label) on
the LHS of a rule, e.g.
\begin{verbatim}
(  
  {Lookup.majorType == jobtitle}  
 ):jobtitle  
 (  
  {TempPerson}  
 ):person  
\end{verbatim}

Nested patterns are also permitted (if correctly
labelled), e.g.
\begin{verbatim}
(  
 (
  {Lookup.majorType == jobtitle}  
 ):jobtitle  

 {TempPerson}  
):person  
\end{verbatim}

\subsection{Context}
Context before or after a pattern can also be indicated by enclosing
it in a set of round brackets. The difference between a pattern to be
annotated and context not to be annotated is simply the presence of a
label, i.e. an unlabelled pattern acts as context. For example, the
following rule would annotate the pattern matched by YEAR if the
words ``in'' or ``by'' preceded it. Note that context cannot be reused
in more than one rule in the same grammar phase, i.e. it is always consumed
if the rule fires. 
\begin{verbatim}
Rule: YearContext1  
  
 ({Token.string == "in"}|  
  {Token.string == "by"}  
 )  
 (YEAR)  
 :date 
\end{verbatim}

\subsection{Macros}
Macros can also be used in the LHS of rules. This means that instead
of expressing the information in the rule, it is specified in a macro,
which can then be called in the rule. The reason for this is simply to
avoid having to repeat the same information in several rules. Macros
can themselves be used inside other macros. A macro used in one phase
of a grammar does not need to be respecified in a later phase, i.e. a
rule may call a macro specified in a previous phase of the same
grammar.

The example below shows a TITLE macro which is then used in a rule.
Conventionally, macros are written in capital letters, though this is
not essential. When used in the rule, a macro must not be enclosed in
curly braces, but only (optionally) in round brackets.

\begin{verbatim}
Macro: TITLE
(
 {Title}
 ({Token.string == "."})?
 ({Title})?
 ({Token.string == "."})?
)

Rule: TitlePerson
(
 (TITLE)
 ({Upper})+
):person
-->
...
\end{verbatim}


\section{Simple Pattern actions}
The RHS of the rule contains information about the annotation to be
given to the pattern. Information about the pattern is transferred from the
LHS of the rule using the label. and annotated with the
entity type (which follows it). Finally, features and their
corresponding values are added to the annotation.  

In the example rule below, the label is ``loc''. The RHS of the rule
is the part which follows the arrow. The label is transferred to the
RHS of the rule and the annotation type ``Location'' is added to the
pattern. The annotation is given two features, ``kind'' and ``rule''
with the values ``city'' and ``GazCity'' respectively. Both of these
features are optional. The first is used to give us more specific
information about the annotation, i.e. that it is a particular kind of
Location. The second is used mainly for debugging purposes, so that we
can keep track of which rule fired. The resulting annotation and its
features and values will all be displayed, together with the string
and the offsets, in the annotations tablein the GATE GUI (assuming
that the grammar has been run on a document and the rule has been
fired).

\begin{verbatim}
Rule: GazCity
 (  
 {Lookup.majorType == city}  
 )  
 :loc -->  
  :loc.Location = {kind="city", rule=``GazCity''} 
\end{verbatim}

\section{Setting the options}

At the beginning of each grammar, several options can be set: 

\textbf{Control} - this defines the method of rule matching. More on this later.
 
\textbf{Debug} - when set to true, if the grammar is running in Appelt mode and
there is more than one possible match, the conflicts will be displayed
in the messages window.

\textbf{Input annotations} must also be defined at the start of each
grammar. If no annotations are defined, the default will be Token,
SpaceToken and Lookup (i.e. only these annotations will be considered
when attempting a match). Every annotation type that is going to be
matched in that grammar phase must be included in the Input set. Any
annotation type that is not defined will be ignored in the pattern
matching. This is very useful in the case of white space. If
SpaceTokens are specified, then every possible space between tokens in
patterns must be described in the rules. If they are not specified,
then this is not necessary, as their presence or absence will be
ignored.


\section{Use of priority}
Each grammar phase has 3 possible control styles: ``brill'', ``first'' and
``appelt''. This is specified at the beginning of the grammar. For named
entity recognition, the appelt style is generally the most
suitable. When writing JAPE grammars for other uses, the other styles
are often useful.

The ``brill'' style means that when more than one rule matches the same region of
the document, they are all fired. The result of this is that a segment
of text could be allocated more than one entity type, and that no
proiority ordering is necessary.

With the ``first'' style, a rule fires for the first match that is
found. This makes it unappropiate for rules that end in "+" or "?" or
"*". Once a match is found, the rule is fired; it does not attempt to
get a longer match (as the other two styles do).

With the ``appelt'' style, only one rule can be fired for the same
pattern, according to a set of priority rules. Priority operates in
the following way.
\begin{enumerate}
\item From all the rules that match a region of the document starting
at some point X, the one which matches the longest region is fired.

\item If more than one rule matches the same region, the one with the
highest priority is fired.
 
\item If there is more than one rule with the same priority, the one
defined last in the grammar is fired.
\end{enumerate}

An optional priority declaration is associated with each rule, which
should be a positive integer.  The higher the number, the greater the
priority. By default (if the priority declaration is missing) all
rules have the priority -1 (i.e. the lowest priority).


\section{Using phases sequentially}
A JAPE grammar consists of a set of sequential phases. The list of
phases is specified (in the order in which they are to be run) in a
file, conventionally named main.jape. When loading the grammar into
GATE, it is only necessary to load this main file -- the phases will
then be loaded automatically. It is, however, possible to omit this
main file, and just load the phases individually, but this is much
more time-consuming. The grammar phases do not need to be located in
the same directory as the main file, but if they are not, the relative
path should be specified for each phase.

One of the main reasons for using a sequence of phases is that a
pattern can only be used once in each phase, but it can be reused in a
later phase. Combined with the fact that priority can only operate
within a single grammar, this can be exploited to help deal with
ambiguity issues.

The solution currently adopted is to write a grammar phase for each
annotation type, or for each combination of similar annotation types,
and to create temporary annotations. These temporary annotations are
accessed by later grammar phases, and can be manipulated as necessary
to resolve ambiguity or to merge consecutive annotations. The temporary
annotations can either be removed later, or left and simply ignored.

Generally, annotations about which we are more certain are created
earlier on. Annotations which are more dubious may be created
temporarily, and then manipulated by later phases as more information
becomes available.

\section{Some tricks}
Although the JAPE language has some limitations as to how rules and
patterns can be expressed, there are some useful tricks to overcome
these problems.

\subsection{Negative operator}
A negative operator cannot be specified as such. One solution to this
    is to create a ``negative rule'' which has higher priority than the
    matching ``positive rule''. The style of matching must be Appelt for
    this to work. To create a negative rule, simply state on the LHS
    of the rule the pattern that should NOT be matched, and on the RHS
    do nothing. In this way, the positive rule cannot be fired if the
    negative pattern matches, and vice versa, which has the same end
    result as using a negative operator. A useful variation for
    developers is to create a dummy annotation on the RHS of the
    negative rule, rather than to do nothing, and to give the dummy
    annotation a rule feature. In this way, it is obvious that the
    negative rule has fired. Alternatively, use Java code on the RHS
    to print a message when the rule fires. An example of a matching
    negative and positive rule follows. Here, we want a rule which
    matches a surname followed by a comma and a set of initials. But
    we want to specify that the initials shouldn't have the POS
    category PRP (personal pronoun). So we specify a negative rule
    that will fire if the PRP category exists, thereby preventing the
    positive rule from firing.

\begin{verbatim}
Rule: NotPersonReverse
Priority: 20 
// we don't want to match ''Jones, I'' 

( {Token.category == NNP}
  {Token.string == ","}
  {Token.category == PRP} 
):label 
--> 
{}
      
Rule:   PersonReverse  
Priority: 5  
// we want to match ``Jones, F.W.''  
   
(  
 {Token.category == NNP}  
 {Token.string == ","}  
 (INITIALS)?  
)  
:person --> 
 :person.Person = {rule = ``PersonReverse''}
\end{verbatim}

\subsection{Matching special characters}
To specify a single or double quote as a string, precede it with a
backslash, e.g. \begin{verbatim}{Token.string=="\""}\end{verbatim}
will match a double quote.  For other special characters, such as
"\$", enclose it in double quotes,
e.g. \begin{verbatim}{Token.category == "PRP$"}\end{verbatim}.


