Contents

Part I
GATE Basics [#]

Chapter 1
Introduction [#]

GATE¹ is an infrastructure for developing and deploying software components that process human language. It is nearly 15 years old and is in active use for all types of computational task involving human language. GATE excels at text analysis of all shapes and sizes. From large corporations to small startups, from €multi-million research consortia to undergraduate projects, our user community is the largest and most diverse of any system of this type, and is spread across all but one of the continents².

GATE is open source free software; users can obtain free support from the user and developer community via GATE.ac.uk or on a commercial basis from our industrial partners. We are the biggest open source language processing project with a development team more than double the size of the largest comparable projects (many of which are integrated with GATE³). More than €5 million has been invested in GATE development⁴; our objective is to make sure that this continues to be money well spent for all GATE’s users.

The GATE family of tools has grown over the years to include a desktop client for developers, a workflow-based web application, a Java library, an architecture and a process. GATE is:

• an IDE, GATE Developer: an integrated development environment⁵ for language processing components bundled with a very widely used Information Extraction system and a comprehensive set of other plugins

• a cloud computing solution for hosted large-scale text processing, GATE Cloud (https://cloud.gate.ac.uk/). See also Chapter 24.

• a web app, GATE Teamware: a collaborative annotation environment for factory-style semantic annotation projects built around a workflow engine and a heavily-optimised backend service infrastructure. See also Chapter 25.

• a multi-paradigm search repository, GATE Mímir, which can be used to index and search over text, annotations, semantic schemas (ontologies), and semantic meta-data (instance data). It allows queries that arbitrarily mix full-text, structural, linguistic and semantic queries and that can scale to terabytes of text. See also Chapter 26.

• a framework, GATE Embedded: an object library optimised for inclusion in diverse applications giving access to all the services used by GATE Developer and more.

• an architecture: a high-level organisational picture of how language processing software composition.

• a process for the creation of robust and maintainable services.

We also develop:

• a wiki/CMS, GATE Wiki (http://gatewiki.sf.net/), mainly to host our own websites and as a testbed for some of our experiments

For more information on the GATE family see http://gate.ac.uk/family/ and also Part IV of this book.

One of our original motivations was to remove the necessity for solving common engineering problems before doing useful research, or re-engineering before deploying research results into applications. Core functions of GATE take care of the lion’s share of the engineering:

• modelling and persistence of specialised data structures

• measurement, evaluation, benchmarking (never believe a computing researcher who hasn’t measured their results in a repeatable and open setting!)

• visualisation and editing of annotations, ontologies, parse trees, etc.

• a finite state transduction language for rapid prototyping and efficient implementation of shallow analysis methods (JAPE)

• extraction of training instances for machine learning

• pluggable machine learning implementations (Weka, SVM Light, ...)

On top of the core functions GATE includes components for diverse language processing tasks, e.g. parsers, morphology, tagging, Information Retrieval tools, Information Extraction components for various languages, and many others. GATE Developer and Embedded are supplied with an Information Extraction system (ANNIE) which has been adapted and evaluated very widely (numerous industrial systems, research systems evaluated in MUC, TREC, ACE, DUC, Pascal, NTCIR, etc.). ANNIE is often used to create RDF or OWL (metadata) for unstructured content (semantic annotation).

GATE version 1 was written in the mid-1990s; at the turn of the new millennium we completely rewrote the system in Java; version 5 was released in June 2009; and version 6 — in November 2010. We believe that GATE is the leading system of its type, but as scientists we have to advise you not to take our word for it; that’s why we’ve measured our software in many of the competitive evaluations over the last decade-and-a-half (MUC, TREC, ACE, DUC and more; see Section 1.4 for details). We invite you to give it a try, to get involved with the GATE community, and to contribute to human language science, engineering and development.

This book describes how to use GATE to develop language processing components, test their performance and deploy them as parts of other applications. In the rest of this chapter:

• Section 1.1 describes the best way to use this book;

• Section 1.2 briefly notes that the context of GATE is applied language processing, or Language Engineering;

• Section 1.3 gives an overview of developing using GATE;

• Section 1.4 lists publications describing GATE performance in evaluations;

• Section 1.5 outlines what is new in the current version of GATE;

• Section 1.6 lists other publications about GATE.

Note: if you don’t see the component you need in this document, or if we mention a component that you can’t see in the software, contact gate-users@lists.sourceforge.net⁶ – various components are developed by our collaborators, who we will be happy to put you in contact with. (Often the process of getting a new component is as simple as typing the URL into GATE Developer; the system will do the rest.)

1.1 How to Use this Text [#]

The material presented in this book ranges from the conceptual (e.g. ‘what is software architecture?’) to practical instructions for programmers (e.g. how to deal with GATE exceptions) and linguists (e.g. how to write a pattern grammar). Furthermore, GATE’s highly extensible nature means that new functionality is constantly being added in the form of new plugins. Important functionality is as likely to be located in a plugin as it is to be integrated into the GATE core. This presents something of an organisational challenge. Our (no doubt imperfect) solution is to divide this book into three parts. Part I covers installation, using the GATE Developer GUI and using ANNIE, as well as providing some background and theory. We recommend the new user to begin with Part I. Part II covers the more advanced of the core GATE functionality; the GATE Embedded API and JAPE pattern language among other things. Part III provides a reference for the numerous plugins that have been created for GATE. Although ANNIE provides a good starting point, the user will soon wish to explore other resources, and so will need to consult this part of the text. We recommend that Part III be used as a reference, to be dipped into as necessary. In Part III, plugins are grouped into broad areas of functionality.

1.2 Context [#]

GATE can be thought of as a Software Architecture for Language Engineering [Cunningham 00].

‘Software Architecture’ is used rather loosely here to mean computer infrastructure for software development, including development environments and frameworks, as well as the more usual use of the term to denote a macro-level organisational structure for software systems [Shaw & Garlan 96].

Language Engineering (LE) may be defined as:

…the discipline or act of engineering software systems that perform tasks involving processing human language. Both the construction process and its outputs are measurable and predictable. The literature of the field relates to both application of relevant scientific results and a body of practice. [Cunningham 99a]

The relevant scientific results in this case are the outputs of Computational Linguistics, Natural Language Processing and Artificial Intelligence in general. Unlike these other disciplines, LE, as an engineering discipline, entails predictability, both of the process of constructing LE-based software and of the performance of that software after its completion and deployment in applications.

Some working definitions:

1. Computational Linguistics (CL): science of language that uses computation as an investigative tool.

2. Natural Language Processing (NLP): science of computation whose subject matter is data structures and algorithms for computer processing of human language.

3. Language Engineering (LE): building NLP systems whose cost and outputs are measurable and predictable.

4. Software Architecture: macro-level organisational principles for families of systems. In this context is also used as infrastructure.

5. Software Architecture for Language Engineering (SALE): software infrastructure, architecture and development tools for applied CL, NLP and LE.

(Of course the practice of these fields is broader and more complex than these definitions.)

In the scientific endeavours of NLP and CL, GATE’s role is to support experimentation. In this context GATE’s significant features include support for automated measurement (see Chapter 10), providing a ‘level playing field’ where results can easily be repeated across different sites and environments, and reducing research overheads in various ways.

1.3 Overview [#]

1.3.1 Developing and Deploying Language Processing Facilities [#]

GATE as an architecture suggests that the elements of software systems that process natural language can usefully be broken down into various types of component, known as resources⁷. Components are reusable software chunks with well-defined interfaces, and are a popular architectural form, used in Sun’s Java Beans and Microsoft’s .Net, for example. GATE components are specialised types of Java Bean, and come in three flavours:

• LanguageResources (LRs) represent entities such as lexicons, corpora or ontologies;

• ProcessingResources (PRs) represent entities that are primarily algorithmic, such as parsers, generators or ngram modellers;

• VisualResources (VRs) represent visualisation and editing components that participate in GUIs.

These definitions can be blurred in practice as necessary.

Collectively, the set of resources integrated with GATE is known as CREOLE: a Collection of REusable Objects for Language Engineering. All the resources are packaged as Java Archive (or ‘JAR’) files, plus some XML configuration data. The JAR and XML files are made available to GATE by putting them on a web server, or simply placing them in the local file space. Section 1.3.2 introduces GATE’s built-in resource set.

When using GATE to develop language processing functionality for an application, the developer uses GATE Developer and GATE Embedded to construct resources of the three types. This may involve programming, or the development of Language Resources such as grammars that are used by existing Processing Resources, or a mixture of both. GATE Developer is used for visualisation of the data structures produced and consumed during processing, and for debugging, performance measurement and so on. For example, figure 1.1 is a screenshot of one of the visualisation tools.

GATE Developer is analogous to systems like Mathematica for Mathematicians, or JBuilder for Java programmers: it provides a convenient graphical environment for research and development of language processing software.

When an appropriate set of resources have been developed, they can then be embedded in the target client application using GATE Embedded. GATE Embedded is supplied as a series of JAR files.⁸ To embed GATE-based language processing facilities in an application, these JAR files are all that is needed, along with JAR files and XML configuration files for the various resources that make up the new facilities.

1.3.2 Built-In Components [#]

GATE includes resources for common LE data structures and algorithms, including documents, corpora and various annotation types, a set of language analysis components for Information Extraction and a range of data visualisation and editing components.

GATE supports documents in a variety of formats including XML, RTF, email, HTML, SGML and plain text. In all cases the format is analysed and converted into a single unified model of annotation. The annotation format is a modified form of the TIPSTER format [Grishman 97] which has been made largely compatible with the Atlas format [Bird & Liberman 99], and uses the now standard mechanism of ‘stand-off markup’. GATE documents, corpora and annotations are stored in databases of various sorts, visualised via the development environment, and accessed at code level via the framework. See Chapter 5 for more details of corpora etc.

A family of Processing Resources for language analysis is included in the shape of ANNIE, A Nearly-New Information Extraction system. These components use finite state techniques to implement various tasks from tokenisation to semantic tagging or verb phrase chunking. All ANNIE components communicate exclusively via GATE’s document and annotation resources. See Chapter 6 for more details. Other CREOLE resources are described in Part III.

1.3.3 Additional Facilities in GATE Developer/Embedded [#]

Three other facilities in GATE deserve special mention:

• JAPE, a Java Annotation Patterns Engine, provides regular-expression based pattern/action rules over annotations – see Chapter 8.

• The ‘annotation diff’ tool in the development environment implements performance metrics such as precision and recall for comparing annotations. Typically a language analysis component developer will mark up some documents by hand and then use these along with the diff tool to automatically measure the performance of the components. See Chapter 10.

• GUK, the GATE Unicode Kit, fills in some of the gaps in the JDK’s⁹ support for Unicode, e.g. by adding input methods for various languages from Urdu to Chinese. See Section 3.11.2 for more details.

1.3.4 An Example [#]

This section gives a very brief example of a typical use of GATE to develop and deploy language processing capabilities in an application, and to generate quantitative results for scientific publication.

Let’s imagine that a developer called Fatima is building an email client¹⁰ for Cyberdyne Systems’ large corporate Intranet. In this application she would like to have a language processing system that automatically spots the names of people in the corporation and transforms them into mailto hyperlinks.

A little investigation shows that GATE’s existing components can be tailored to this purpose. Fatima starts up GATE Developer, and creates a new document containing some example emails. She then loads some processing resources that will do named-entity recognition (a tokeniser, gazetteer and semantic tagger), and creates an application to run these components on the document in sequence. Having processed the emails, she can see the results in one of several viewers for annotations.

The GATE components are a decent start, but they need to be altered to deal specially with people from Cyberdyne’s personnel database. Therefore Fatima creates new ‘cyber-’ versions of the gazetteer and semantic tagger resources, using the ‘bootstrap’ tool. This tool creates a directory structure on disk that has some Java stub code, a Makefile and an XML configuration file. After several hours struggling with badly written documentation, Fatima manages to compile the stubs and create a JAR file containing the new resources. She tells GATE Developer the URL of these files¹¹, and the system then allows her to load them in the same way that she loaded the built-in resources earlier on.

Fatima then creates a second copy of the email document, and uses the annotation editing facilities to mark up the results that she would like to see her system producing. She saves this and the version that she ran GATE on into her serial datastore. From now on she can follow this routine:

1. Run her application on the email test corpus.

2. Check the performance of the system by running the ‘annotation diff’ tool to compare her manual results with the system’s results. This gives her both percentage accuracy figures and a graphical display of the differences between the machine and human outputs.

3. Make edits to the code, pattern grammars or gazetteer lists in her resources, and recompile where necessary.

4. Tell GATE Developer to re-initialise the resources.

5. Go to 1.

To make the alterations that she requires, Fatima re-implements the ANNIE gazetteer so that it regenerates itself from the local personnel data. She then alters the pattern grammar in the semantic tagger to prioritise recognition of names from that source. This latter job involves learning the JAPE language (see Chapter 8), but as this is based on regular expressions it isn’t too difficult.

Eventually the system is running nicely, and her accuracy is 93% (there are still some problem cases, e.g. when people use nicknames, but the performance is good enough for production use). Now Fatima stops using GATE Developer and works instead on embedding the new components in her email application using GATE Embedded. This application is written in Java, so embedding is very easy¹²: the GATE JAR files are added to the project CLASSPATH, the new components are placed on a web server, and with a little code to do initialisation, loading of components and so on, the job is finished in half a day – the code to talk to GATE takes up only around 150 lines of the eventual application, most of which is just copied from the example in the sheffield.examples.StandAloneAnnie class.

Because Fatima is worried about Cyberdyne’s unethical policy of developing Skynet to help the large corporates of the West strengthen their strangle-hold over the World, she wants to get a job as an academic instead (so that her conscience will only have to cope with the torture of students, as opposed to humanity). She takes the accuracy measures that she has attained for her system and writes a paper for the Journal of Nasturtium Logarithm Incitement describing the approach used and the results obtained. Because she used GATE for development, she can cite the repeatability of her experiments and offer access to example binary versions of her software by putting them on an external web server.

And everybody lived happily ever after.

1.4 Some Evaluations [#]

This section contains an incomplete list of publications describing systems that used GATE in competitive quantitative evaluation programmes. These programmes have had a significant impact on the language processing field and the widespread presence of GATE is some measure of the maturity of the system and of our understanding of its likely performance on diverse text processing tasks.

[Li et al. 07d]

describes the performance of an SVM-based learning system in the NTCIR-6 Patent Retrieval Task. The system achieved the best result on two of three measures used in the task evaluation, namely the R-Precision and F-measure. The system obtained close to the best result on the remaining measure (A-Precision).

[Saggion 07]

describes a cross-source coreference resolution system based on semantic clustering. It uses GATE for information extraction and the SUMMA system to create summaries and semantic representations of documents. One system configuration ranked 4th in the Web People Search 2007 evaluation.

[Saggion 06]

describes a cross-lingual summarization system which uses SUMMA components and the Arabic plugin available in GATE to produce summaries in English from a mixture of English and Arabic documents.

Open-Domain Question Answering:

The University of Sheffield has a long history of research into open-domain question answering. GATE has formed the basis of much of this research resulting in systems which have ranked highly during independent evaluations since 1999. The first successful question answering system developed at the University of Sheffield was evaluated as part of TREC 8 and used the LaSIE information extraction system (the forerunner of ANNIE) which was distributed with GATE [Humphreys et al. 99]. Further research was reported in [Scott & Gaizauskas. 00], [Greenwood et al. 02], [Gaizauskas et al. 03], [Gaizauskas et al. 04] and [Gaizauskas et al. 05]. In 2004 the system was ranked 9th out of 28 participating groups.

[Saggion 04]

describes techniques for answering definition questions. The system uses definition patterns manually implemented in GATE as well as learned JAPE patterns induced from a corpus. In 2004, the system was ranked 4th in the TREC/QA evaluations.

[Saggion & Gaizauskas 04b]

describes a multidocument summarization system implemented using summarization components compatible with GATE (the SUMMA system). The system was ranked 2nd in the Document Understanding Evaluation programmes.

[Maynard et al. 03e] and [Maynard et al. 03d]

describe participation in the TIDES surprise language program. ANNIE was adapted to Cebuano with four person days of effort, and achieved an F-measure of 77.5%. Unfortunately, ours was the only system participating!

[Maynard et al. 02b] and [Maynard et al. 03b]

describe results obtained on systems designed for the ACE task (Automatic Content Extraction). Although a comparison to other participating systems cannot be revealed due to the stipulations of ACE, results show 82%-86% precision and recall.

[Humphreys et al. 98]

describes the LaSIE-II system used in MUC-7.

[Gaizauskas et al. 95]

describes the LaSIE-II system used in MUC-6.

1.5 Recent Changes [#]

This section details recent changes made to GATE. Appendix A provides a complete change log.

It was brought to our attention that in versions 9.0.1 and below there was a very small chance that the GUI action “Export for GATE Cloud” could be compromised. This would have required malicious code to be running locally on the machine; either by another user on a multi-user machine or because the computer had already been compromised. This issue only occurred within the GUI action and did not affect API use of the gate-core Maven artifact. Note that no known exploits exist for this issue, and we do not know for certain that the code could be exploited. If, however, you are at all concerned then we suggest you regenerate any packaged applications using a recent version of GATE Developer; at minimum 9.2-SNAPSHOT built on or after the 10th of August 2022.

1.5.1 Version 9.0.1 (March 2021) [#]

GATE Developer 9.0.1 is a bugfix release – the only change is to the way URL redirects are handled when loading a document. Support for following redirects from http to https was added in 9.0 which, while correct, broke the way URLs were used within GCP. This release fixes that bug and adds some additional security checking to the redirect handling.

1.5.2 Version 9.0 (February 2021) [#]

Whilst the majority of changes in GATE Developer 9.0 are small a number of them change default behaviour (in the UI or API) hence the change in version number. These changes include:

• We now recommend users install a 64 bit version of Java whenever possible. This seems to be especially important on Windows.

• We now default to assuming documents are UTF-8 encoded unless you specify otherwise. In previous versions if no encoding was specified GATE would use the default platform encoding, but this seemed to cause more problems than it solved (especially for Windows users). If you want the old behaviour then ensure the encoding parameter is set to the empty string when creating a document.

• GATE uses a library called XStream for saving and loading GATE XML documents and applications. This allows us to store features of any Java type, but that can be abused by maliciously crafted files. In general use this is unlikely to be a problem, but in situations where GATE may be used as part of a service with no way of vetting input files it could present a serious security threat. XStream now offers a security framework to restrict the types of objects that can be loaded/saved. This can work either by allowing only specific types or by preventing specific types from being used. As we often do not know in advance what features might be used we have opted to use a minimal blacklist as the default security setting. This blocks the Java classes known to be exploitable. This can be further configured via calls to Gate.setXStreamSecurity() and we strongly encourage developers who depend on gate-core within larger applications to configure this based on their specific use cases.

• Developers wishing to build GATE from source need to use Maven v3.6.0 or above.

• Previous versions of GATE used Log4J for some of the logging. This was problematic when using gate-core as a dependency in larger projects and was awkward to configure properly. In this release we’ve switched to using SLF4J allowing the actual logging back-end to be configured independently. Plugins and code compiled against previous versions of GATE should work with the new release without change (we include the log4j-over-slf4j bridge as a dependency), although Log4J specific methods within gate-core have been deprecated and may be removed in a future release.

Many bugs have been fixed and documentation improved, in particular:

• the Twitter plugin has been improved to make better use of the information provided by Twitter within a JSON Tweet object. The Hashtag tokenizer has been updated to provide a tokenized feature to make grouping semantically similar hashtags easier. Lots of other minor improvements and efficiency changes have been made throughout the rest of the TwitIE pipelines.

• the ANNIE gazetteers have been updated to better support different ways of referring to countries and a blacklist option to prevent things being wrongly annotated.

• A new addition to the JAPE syntax allows you to copy all features from a matched annotation to the new annotation being created

• the Format_CSV plugin now allows the document cell to be interpreted as being a URL pointing to the document to load rather than the contents of the document. See Section 23.33 for more details.

1.5.3 Version 8.6.1 (January 2020) [#]

GATE Developer 8.6.1 is a bugfix release – the only change is to adjust for the fact that the Central Maven repository has been switched from http to https.

1.5.4 Version 8.6 (June 2019) [#]

GATE Developer 8.6 is mainly a maintenance and stability release, but there are some important new features, in particular around the processing of Twitter data:

• The Format_Twitter plugin can now correctly handle extended 280 character tweets and the latest Twitter JSON format. See Section 17.2 for full details.

• The new Format_JSON plugin provides import/export support for GATE JSON. This is essentially the old style Twitter format, but it no longer needs to track changes to the Twitter JSON format so should be more suitable for long term storage of GATE documents as JSON files. See Section 23.30 for more details. This plugin makes use of a new mechanism whereby document format parsers can take parameters via the document MIME type, which may be useful to third party formats too.

Many bugs have been fixed and documentation improved, in particular:

• The plugin loading mechanism now properly respects the user’s Maven settings.xml:

  • HTTP proxy and “mirror” repository settings now work properly, including authentication. Also plugin resolution will now use the system proxy (if there is one) by default if there is no proxy specified in the Maven settings.

  • The “offline” setting is respected, and will prevent GATE from trying to fetch plugins from remote repositories altogether – for this to work, all the plugins you want to use must already be cached locally, or you can use “Export for GATE Cloud” to make a self-contained copy of an application including all its plugins.

• Upgraded many dependencies including Tika and Jackson to avoid known security bugs in the previous versions.

• Documentation improvements for the Kea plugin, the Corpus QA and annotation diff tools, and the default GATE XML and inline XML formats (section 3.9.1)

• For plugin developers, the standard plugin testing framework generates a report detailing all the plugin-to-plugin dependencies, including those that are only expressed in the plugin’s example saved applications (section 7.12.1).

Some obsolete plugins have been removed (Websphinx web crawler, which depends on an unmaintained library, and the RASP parser, whose external binary is no longer available for modern operating systems), and there are many smaller bug fixes and improvements.

Note: following changes to Oracle’s JDK licensing scheme, we now recommend running GATE using the freely-available OpenJDK. The AdoptOpenJDK project offers simple installers for all major platforms, and major Linux distributions such as Ubuntu and CentOS offer OpenJDK packages as standard. See section 2.2 for full installation instructions.

1.5.5 Version 8.5.1 (June 2018) [#]

Version 8.5.1 is a minor release to fix a few critical bugs in 8.5:

• Fixed an exception that prevented the ANNIC search GUI from opening.

• Fixed a problem with “Export for GATE Cloud” that meant some resources were not getting included in the output ZIP file.

• Fixed the XML schema in the gate-spring library.

1.5.6 Version 8.5 (May 2018) [#]

GATE Developer and Embedded 8.5 introduces a number of significant internal changes to the way plugins are managed, but with the exception of the plugin manager most users will not see significant changes in the way they use GATE.

• The GATE plugins are no longer bundled with the GATE Developer distribution, instead each plugin is downloaded from a repository at runtime, the first time it is used. This means the distribution is much smaller than previous versions.

• Most plugins are now distributed as a single JAR file through the Java-standard “Central Repository”, and resource files such as gazetteers and JAPE grammars are bundled inside the plugin JAR rather than being separate files on disk. If you want to modify the resources of a plugin then GATE provides a tool to extract an editable copy of the files from a plugin onto your disk – it is no longer possible to edit plugin grammars in place.

• This makes dependencies between plugins much easier to manage – a plugin can specify its dependencies declaratively by name and version number rather than by fragile relative paths between plugin directories.

GATE 8.5 remains backwards compatible with existing third-party plugins, though we encourage you to convert your plugins to the new style where possible.

Further details on these changes can be found in sections 3.5 (the plugin manager in GATE Developer), 7.3 (loading plugins via the GATE Embedded API), 7.12 (creating a new plugin from scratch), and 7.20 (converting an existing plugin to the new style).

If you have an existing saved application from GATE version 8.4.1 or earlier it will be necessary to “upgrade” it to use the new core plugins. An upgrade tool is provided on the “Tools” menu of GATE Developer, and is described in section Section 3.9.5.

For developers

As part of this release, GATE development has moved from SourceForge to GitHub – bug reports, patches and feature requests should now use the GitHub issue tracker as described in section 12.1.

1.6 Further Reading [#]

Lots of documentation lives on the GATE web site, including:

• GATE online tutorials;

• the main system documentation tree;

• JavaDoc API documentation;

• source code (at GitHub);

For more details about Sheffield University’s work in human language processing see the NLP group pages or A Definition and Short History of Language Engineering ([Cunningham 99a]). For more details about Information Extraction see IE, a User Guide or the GATE IE pages.

A list of publications on GATE and projects that use it (some of which are available on-line from http://gate.ac.uk/gate/doc/papers.html):

2010

[Bontcheva et al. 10]

describes the Teamware web-based collaborative annotation environment, emphasising the different roles that users play in the corpus annotation process.

[Damljanovic 10]

presents the use of GATE in the development of controlled natural language interfaces. There is other related work by Damljanovic, Agatonovic, and Cunningham on using GATE to build natural language interfaces for quering ontologies.

[Aswani & Gaizauskas 10]

discusses the use of GATE to process South Asian languages (Hindi and Gujarati).

2009

[Saggion & Funk 09]

focuses in detail on the use of GATE for mining opinions and facts for business intelligence gathering from web content.

[Aswani & Gaizauskas 09]

presents in more detail the text alignment component of GATE.

[Bontcheva et al. 09]

is the ‘Human Language Technologies’ chapter of ‘Semantic Knowledge Management’ (John Davies, Marko Grobelnik and Dunja Mladenić eds.)

[Damljanovic et al. 09]

discusses the use of semantic annotation for software engineering, as part of the TAO research project.

[Laclavik & Maynard 09]

reviews the current state of the art in email processing and communication research, focusing on the roles played by email in information management, and commercial and research efforts to integrate a semantic-based approach to email.

[Li et al. 09]

investigates two techniques for making SVMs more suitable for language learning tasks. Firstly, an SVM with uneven margins (SVMUM) is proposed to deal with the problem of imbalanced training data. Secondly, SVM active learning is employed in order to alleviate the difficulty in obtaining labelled training data. The algorithms are presented and evaluated on several Information Extraction (IE) tasks.

2008

[Agatonovic et al. 08]

presents our approach to automatic patent enrichment, tested in large-scale, parallel experiments on USPTO and EPO documents.

[Damljanovic et al. 08]

presents Question-based Interface to Ontologies (QuestIO) - a tool for querying ontologies using unconstrained language-based queries.

[Damljanovic & Bontcheva 08]

presents a semantic-based prototype that is made for an open-source software engineering project with the goal of exploring methods for assisting open-source developers and software users to learn and maintain the system without major effort.

[Della Valle et al. 08]

presents ServiceFinder.

[Li & Cunningham 08]

describes our SVM-based system and several techniques we developed successfully to adapt SVM for the specific features of the F-term patent classification task.

[Li & Bontcheva 08]

reviews the recent developments in applying geometric and quantum mechanics methods for information retrieval and natural language processing.

[Maynard 08]

investigates the state of the art in automatic textual annotation tools, and examines the extent to which they are ready for use in the real world.

[Maynard et al. 08a]

discusses methods of measuring the performance of ontology-based information extraction systems, focusing particularly on the Balanced Distance Metric (BDM), a new metric we have proposed which aims to take into account the more flexible nature of ontologically-based applications.

[Maynard et al. 08b]

investigates NLP techniques for ontology population, using a combination of rule-based approaches and machine learning.

[Tablan et al. 08]

presents the QuestIO system – a natural language interface for accessing structured information, that is domain independent and easy to use without training.

2007

[Funk et al. 07a]

describes an ontologically based approach to multi-source, multilingual information extraction.

[Funk et al. 07b]

presents a controlled language for ontology editing and a software implementation, based partly on standard NLP tools, for processing that language and manipulating an ontology.

[Maynard et al. 07a]

proposes a methodology to capture (1) the evolution of metadata induced by changes to the ontologies, and (2) the evolution of the ontology induced by changes to the underlying metadata.

[Maynard et al. 07b]

describes the development of a system for content mining using domain ontologies, which enables the extraction of relevant information to be fed into models for analysis of financial and operational risk and other business intelligence applications such as company intelligence, by means of the XBRL standard.

[Saggion 07]

describes experiments for the cross-document coreference task in SemEval 2007. Our cross-document coreference system uses an in-house agglomerative clustering implementation to group documents referring to the same entity.

[Saggion et al. 07]

describes the application of ontology-based extraction and merging in the context of a practical e-business application for the EU MUSING Project where the goal is to gather international company intelligence and country/region information.

[Li et al. 07a]

introduces a hierarchical learning approach for IE, which uses the target ontology as an essential part of the extraction process, by taking into account the relations between concepts.

[Li et al. 07b]

proposes some new evaluation measures based on relations among classification labels, which can be seen as the label relation sensitive version of important measures such as averaged precision and F-measure, and presents the results of applying the new evaluation measures to all submitted runs for the NTCIR-6 F-term patent classification task.

[Li et al. 07c]

describes the algorithms and linguistic features used in our participating system for the opinion analysis pilot task at NTCIR-6.

[Li et al. 07d]

describes our SVM-based system and the techniques we used to adapt the approach for the specifics of the F-term patent classification subtask at NTCIR-6 Patent Retrieval Task.

[Li & Shawe-Taylor 07]

studies Japanese-English cross-language patent retrieval using Kernel Canonical Correlation Analysis (KCCA), a method of correlating linear relationships between two variables in kernel defined feature spaces.

2006

[Aswani et al. 06]

(Proceedings of the 5th International Semantic Web Conference (ISWC2006)) In this paper the problem of disambiguating author instances in ontology is addressed. We describe a web-based approach that uses various features such as publication titles, abstract, initials and co-authorship information.

[Bontcheva et al. 06a]

‘Semantic Annotation and Human Language Technology’, contribution to ‘Semantic Web Technology: Trends and Research’ (Davies, Studer and Warren, eds.)

[Bontcheva et al. 06b]

‘Semantic Information Access’, contribution to ‘Semantic Web Technology: Trends and Research’ (Davies, Studer and Warren, eds.)

[Bontcheva & Sabou 06]

presents an ontology learning approach that 1) exploits a range of information sources associated with software projects and 2) relies on techniques that are portable across application domains.

[Davis et al. 06]

describes work in progress concerning the application of Controlled Language Information Extraction - CLIE to a Personal Semantic Wiki - Semper- Wiki, the goal being to permit users who have no specialist knowledge in ontology tools or languages to semi-automatically annotate their respective personal Wiki pages.

[Li & Shawe-Taylor 06]

studies a machine learning algorithm based on KCCA for cross-language information retrieval. The algorithm is applied to Japanese-English cross-language information retrieval.

[Maynard et al. 06]

discusses existing evaluation metrics, and proposes a new method for evaluating the ontology population task, which is general enough to be used in a variety of situation, yet more precise than many current metrics.

[Tablan et al. 06b]

describes an approach that allows users to create and edit ontologies simply by using a restricted version of the English language. The controlled language described is based on an open vocabulary and a restricted set of grammatical constructs.

[Tablan et al. 06a]

describes the creation of linguistic analysis and corpus search tools for Sumerian, as part of the development of the ETCSL.

[Wang et al. 06]

proposes an SVM based approach to hierarchical relation extraction, using features derived automatically from a number of GATE-based open-source language processing tools.

2005

[Aswani et al. 05]

(Proceedings of Fifth International Conference on Recent Advances in Natural Language Processing (RANLP2005)) It is a full-featured annotation indexing and search engine, developed as a part of the GATE. It is powered with Apache Lucene technology and indexes a variety of documents supported by the GATE.

[Bontcheva 05]

presents the ONTOSUM system which uses Natural Language Generation (NLG) techniques to produce textual summaries from Semantic Web ontologies.

[Cunningham 05]

is an overview of the field of Information Extraction for the 2nd Edition of the Encyclopaedia of Language and Linguistics.

[Cunningham & Bontcheva 05]

is an overview of the field of Software Architecture for Language Engineering for the 2nd Edition of the Encyclopaedia of Language and Linguistics.

[Dowman et al. 05a]

(Euro Interactive Television Conference Paper) A system which can use material from the Internet to augment television news broadcasts.

[Dowman et al. 05b]

(World Wide Web Conference Paper) The Web is used to assist the annotation and indexing of broadcast news.

[Dowman et al. 05c]

(Second European Semantic Web Conference Paper) A system that semantically annotates television news broadcasts using news websites as a resource to aid in the annotation process.

[Li et al. 05c]

(Proceedings of Sheffield Machine Learning Workshop) describe an SVM based IE system which uses the SVM with uneven margins as learning component and the GATE as NLP processing module.

[Li et al. 05a]

(Proceedings of Ninth Conference on Computational Natural Language Learning (CoNLL-2005)) uses the uneven margins versions of two popular learning algorithms SVM and Perceptron for IE to deal with the imbalanced classification problems derived from IE.

[Li et al. 05b]

(Proceedings of Fourth SIGHAN Workshop on Chinese Language processing (Sighan-05)) a system for Chinese word segmentation based on Perceptron learning, a simple, fast and effective learning algorithm.

[Polajnar et al. 05]

(University of Sheffield-Research Memorandum CS-05-10) User-Friendly Ontology Authoring Using a Controlled Language.

[Saggion & Gaizauskas 05]

describes experiments on content selection for producing biographical summaries from multiple documents.

[Ursu et al. 05]

(Proceedings of the 2nd European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies (EWIMT 2005))Digital Media Preservation and Access through Semantically Enhanced Web-Annotation.

[Wang et al. 05]

(Proceedings of the 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI 2005)) Extracting a Domain Ontology from Linguistic Resource Based on Relatedness Measurements.

2004

[Bontcheva 04]

(LREC 2004) describes lexical and ontological resources in GATE used for Natural Language Generation.

[Bontcheva et al. 04]

(JNLE) discusses developments in GATE in the early naughties.

[Cunningham & Scott 04a]

(JNLE) is the introduction to the above collection.

[Cunningham & Scott 04b]

(JNLE) is a collection of papers covering many important areas of Software Architecture for Language Engineering.

[Dimitrov et al. 04]

(Anaphora Processing) gives a lightweight method for named entity coreference resolution.

[Li et al. 04]

(Machine Learning Workshop 2004) describes an SVM based learning algorithm for IE using GATE.

[Maynard et al. 04a]

(LREC 2004) presents algorithms for the automatic induction of gazetteer lists from multi-language data.

[Maynard et al. 04b]

(ESWS 2004) discusses ontology-based IE in the hTechSight project.

[Maynard et al. 04c]

(AIMSA 2004) presents automatic creation and monitoring of semantic metadata in a dynamic knowledge portal.

[Saggion & Gaizauskas 04a]

describes an approach to mining definitions.

[Saggion & Gaizauskas 04b]

describes a sentence extraction system that produces two sorts of multi-document summaries; a general-purpose summary of a cluster of related documents and an entity-based summary of documents related to a particular person.

[Wood et al. 04]

(NLDB 2004) looks at ontology-based IE from parallel texts.

2003

[Bontcheva et al. 03]

(NLPXML-2003) looks at GATE for the semantic web.

[Cunningham et al. 03]

(Corpus Linguistics 2003) describes GATE as a tool for collaborative corpus annotation.

[Kiryakov 03]

(Technical Report) discusses semantic web technology in the context of multimedia indexing and search.

[Manov et al. 03]

(HLT-NAACL 2003) describes experiments with geographic knowledge for IE.

[Maynard et al. 03a]

(EACL 2003) looks at the distinction between information and content extraction.

[Maynard et al. 03c]

(Recent Advances in Natural Language Processing 2003) looks at semantics and named-entity extraction.

[Maynard et al. 03e]

(ACL Workshop 2003) describes NE extraction without training data on a language you don’t speak (!).

[Saggion et al. 03a]

(EACL 2003) discusses robust, generic and query-based summarisation.

[Saggion et al. 03b]

(Data and Knowledge Engineering) discusses multimedia indexing and search from multisource multilingual data.

[Saggion et al. 03c]

(EACL 2003) discusses event co-reference in the MUMIS project.

[Tablan et al. 03]

(HLT-NAACL 2003) presents the OLLIE on-line learning for IE system.

[Wood et al. 03]

(Recent Advances in Natural Language Processing 2003) discusses using parallel texts to improve IE recall.

2002

[Baker et al. 02]

(LREC 2002) report results from the EMILLE Indic languages corpus collection and processing project.

[Bontcheva et al. 02a]

(ACl 2002 Workshop) describes how GATE can be used as an environment for teaching NLP, with examples of and ideas for future student projects developed within GATE.

[Bontcheva et al. 02b]

(NLIS 2002) discusses how GATE can be used to create HLT modules for use in information systems.

[Bontcheva et al. 02c], [Dimitrov 02a] and [Dimitrov 02b]

(TALN 2002, DAARC 2002, MSc thesis) describe the shallow named entity coreference modules in GATE: the orthomatcher which resolves pronominal coreference, and the pronoun resolution module.

[Cunningham 02]

(Computers and the Humanities) describes the philosophy and motivation behind the system, describes GATE version 1 and how well it lived up to its design brief.

[Cunningham et al. 02]

(ACL 2002) describes the GATE framework and graphical development environment as a tool for robust NLP applications.

[Dimitrov 02a, Dimitrov et al. 02]

(DAARC 2002, MSc thesis) discuss lightweight coreference methods.

[Lal 02]

(Master Thesis) looks at text summarisation using GATE.

[Lal & Ruger 02]

(ACL 2002) looks at text summarisation using GATE.

[Maynard et al. 02a]

(ACL 2002 Summarisation Workshop) describes using GATE to build a portable IE-based summarisation system in the domain of health and safety.

[Maynard et al. 02c]

(AIMSA 2002) describes the adaptation of the core ANNIE modules within GATE to the ACE (Automatic Content Extraction) tasks.

[Maynard et al. 02d]

(Nordic Language Technology) describes various Named Entity recognition projects developed at Sheffield using GATE.

[Maynard et al. 02e]

(JNLE) describes robustness and predictability in LE systems, and presents GATE as an example of a system which contributes to robustness and to low overhead systems development.

[Pastra et al. 02]

(LREC 2002) discusses the feasibility of grammar reuse in applications using ANNIE modules.

[Saggion et al. 02b] and [Saggion et al. 02a]

(LREC 2002, SPLPT 2002) describes how ANNIE modules have been adapted to extract information for indexing multimedia material.

[Tablan et al. 02]

(LREC 2002) describes GATE’s enhanced Unicode support.

Older than 2002

[Maynard et al. 01]

(RANLP 2001) discusses a project using ANNIE for named-entity recognition across wide varieties of text type and genre.

[Bontcheva et al. 00] and [Brugman et al. 99]

(COLING 2000, technical report) describe a prototype of GATE version 2 that integrated with the EUDICO multimedia markup tool from the Max Planck Institute.

[Cunningham 00]

(PhD thesis) defines the field of Software Architecture for Language Engineering, reviews previous work in the area, presents a requirements analysis for such systems (which was used as the basis for designing GATE versions 2 and 3), and evaluates the strengths and weaknesses of GATE version 1.

[Cunningham et al. 00a], [Cunningham et al. 98a] and [Peters et al. 98]

(OntoLex 2000, LREC 1998) presents GATE’s model of Language Resources, their access and distribution.

[Cunningham et al. 00b]

(LREC 2000) taxonomises Language Engineering components and discusses the requirements analysis for GATE version 2.

[Cunningham et al. 00c] and [Cunningham et al. 99]

(COLING 2000, AISB 1999) summarise experiences with GATE version 1.

[Cunningham et al. 00d] and [Cunningham 99b]

(technical reports) document early versions of JAPE (superseded by the present document).

[Gambäck & Olsson 00]

(LREC 2000) discusses experiences in the Svensk project, which used GATE version 1 to develop a reusable toolbox of Swedish language processing components.

[Maynard et al. 00]

(technical report) surveys users of GATE up to mid-2000.

[McEnery et al. 00]

(Vivek) presents the EMILLE project in the context of which GATE’s Unicode support for Indic languages has been developed.

[Cunningham 99a]

(JNLE) reviewed and synthesised definitions of Language Engineering.

[Stevenson et al. 98] and [Cunningham et al. 98b]

(ECAI 1998, NeMLaP 1998) report work on implementing a word sense tagger in GATE version 1.

[Cunningham et al. 97b]

(ANLP 1997) presents motivation for GATE and GATE-like infrastructural systems for Language Engineering.

[Cunningham et al. 96a]

(manual) was the guide to developing CREOLE components for GATE version 1.

[Cunningham et al. 96b]

(TIPSTER) discusses a selection of projects in Sheffield using GATE version 1 and the TIPSTER architecture it implemented.

[Cunningham et al. 96c, Cunningham et al. 96d, Cunningham et al. 95]

(COLING 1996, AISB Workshop 1996, technical report) report early work on GATE version 1.

[Gaizauskas et al. 96a]

(manual) was the user guide for GATE version 1.

[Gaizauskas et al. 96b, Cunningham et al. 97a, Cunningham et al. 96e]

(ICTAI 1996, TIPSTER 1997, NeMLaP 1996) report work on GATE version 1.

[Humphreys et al. 96]

(manual) describes the language processing components distributed with GATE version 1.

[Cunningham 94, Cunningham et al. 94]

(NeMLaP 1994, technical report) argue that software engineering issues such as reuse, and framework construction, are important for language processing R&D.

Chapter 2
Installing and Running GATE [#]

2.1 Downloading GATE [#]

To download GATE point your web browser at http://gate.ac.uk/download/.

2.2 Installing and Running GATE [#]

GATE will run anywhere that supports Java 8 or later, including Linux, Mac OS X and Windows platforms. We don’t run tests on other platforms, but have had reports of successful installs elsewhere.

We recommend using OpenJDK 1.8 (or higher). This is widely available from GNU/Linux package repositories. The AdoptOpenJDK website provides packages for various operating systems, and is particularly suitable for Windows users. Mac users should install the JDK (not just the JRE).

Note that wherever possible you should install the 64 bit version of Java as 32 bit versions can have issues with the amount of memory available for GATE to use.

2.2.1 The Easy Way [#]

The easy way to install is to use the installer (created using the excellent IzPack). Download the installer (.exe for Windows, .jar for other platforms) and follow the instructions it gives you. Once the installation is complete, you can start GATE Developer using gate.exe (Windows) or GATE.app (Mac) in the top-level installation directory, on Linux and other platforms use gate.sh in the bin directory (see section 2.2.4).

2.2.2 The Hard Way (1) [#]

• Download and unpack the ZIP distribution, creating a directory containing jar files and scripts.

• To run GATE Developer:

  • on Windows, use the the ‘gate.exe’ file;

  • on UNIX/Linux use ‘bin/gate.sh’.

  • on Mac use ‘GATE.app’ – if running from a terminal you can keep GATE in the foreground using GATE.app/Contents/MacOS/GATE or bin/gate.sh

• To embed GATE as a library (GATE Embedded), put the JAR files in the lib folder onto your application’s classpath. Alternatively you can use a dependency manager to download GATE and its dependencies from the Central Repository by declaring a dependency on the appropriate version of group ID uk.ac.gate and artifact ID gate-core (see section 2.6.1).

2.2.3 The Hard Way (2): Git [#]

The GATE code is maintained in a set of repositories on GitHub. The main repository for GATE Developer and Embedded is gate-core, and each plugin has its own repository (typically with a name beginning gateplugin-).

All the modules (gate-core and the plugins) are built using Apache Maven version 3.5.2 or later. Clone the appropriate repository, checkout the relevant branch (“master” is the latest snapshot version), and build the code using mvn install

See section 2.6 for more details.

2.2.4 Running GATE Developer on Unix/Linux [#]

The script gate.sh in the directory bin of your installation (or distro/bin if you are building from source) can be used to start GATE Developer. You can run this script by entering its full path in a terminal or by adding the bin directory to your binary path. In addition you can also add a symbolic link to this script in any directory that already is in your binary path.

If gate.sh is invoked without parameters, GATE Developer will use the files ~/.gate.xml and ~/.gate.session to store session and configuration data. Alternately you can run gate.sh with the following parameters:

-h

show usage information

-ld

create or use the files .gate.session and .gate.xml in the current directory as the session and configuration files. If option -dc DIR occurs before this option, the file .gate.session is created from DIR/default.session if it does not already exist and the file .gate.xml is created from DIR/default.xml if it does not already exist.

-ln NAME

create or use NAME.session and NAME.xml in the current directory as the session and configuration files. If option -dc DIR occurs before this option, the file NAME.session is created from DIR/default.session if it does not already exist and the file DIR.xml is created from DIR/default.xml if it does not already exist.

-ll FILE

use the file specified to configure the logback logger of Gate Developer. Note that if this is not an absolute path and the name is identical to logback.xml then the default file on the classpath, ${GATE_HOME}/bin/logback.xml is is still used.

-rh LOCATION

set the resources home directory to the LOCATION provided. If a resources home location is provided, the URLs in a saved application are saved relative to this location instead of relative to the application state file (see section 3.9.3). This is equivalent to setting the property gate.user.resourceshome to this location.

-d URL

loads the CREOLE plugin at the given URL during the start-up process.

-i FILE

uses the specified file as the site configuration.

-dc DIR

copy default.xml and/or default.session from the directory DIR when creating a new config or session file. This option works only together with either the -ln, -ll or -tmp option and must occur before -ln, -ll or -tmp. An existing config or session file is used, but if it does not exist, the file from the given directory is copied to create the file instead of using an empty/default file.

-tmp

creates temporary configuration and session files in the current directory, optionally copying default.xml and default.session from the directory specified with a -dc DIR option that occurs before it. After GATE exits, those session and config files are removed.

all other parameters

are passed on to the java command. This can be used to e.g. set properties using the java option -D. For example to set the maximum amount of heap memory to be used when running GATE to 6000M, you can add -Xmx6000m as a parameter. In order to change the default encoding used by GATE to UTF-8 add -Dfile.encoding=utf-8 as a parameter. To specify a log4j configuration file add something like
-Dlog4j.configuration=file:///home/myuser/log4jconfig.properties.

Running GATE Developer with either the -ld or the -ln option from different directories is useful to keep several projects separate and can be used to run multiple instances of GATE Developer (or even different versions of GATE Developer) in succession or even simultanously without the configuration files getting mixed up between them.

2.3 Using System Properties with GATE [#]

During initialisation, GATE reads several Java system properties in order to decide where to find its configuration files.

Here is a list of the properties used, their default values and their meanings:

gate.site.config

points to the location of the configuration file containing the site-wide options. If not set no site config will be used.

gate.user.config

points to the file containing the user’s options. If not specified, or if the specified file does not exist at startup time, the default value of gate.xml (.gate.xml on Unix platforms) in the user’s home directory is used.

gate.user.session

points to the file containing the user’s saved session. If not specified, the default value of gate.session (.gate.session on Unix) in the user’s home directory is used. When starting up GATE Developer, the session is reloaded from this file if it exists, and when exiting GATE Developer the session is saved to this file (unless the user has disabled ‘save session on exit’ in the configuration dialog). The session is not used when using GATE Embedded.

gate.user.filechooser.defaultdir

sets the default directory to be shown in the file chooser of GATE Developer to the specified directory instead of the user’s operating-system specific default directory.

gate.builtin.creole.dir

is a URL pointing to the location of GATE’s built-in CREOLE directory. This is the location of the creole.xml file that defines the fundamental GATE resource types, such as documents, document format handlers, controllers and the basic visual resources that make up GATE. The default points to a location inside gate.jar and should not generally need to be overridden.

When using GATE Embedded, you can set the values for these properties before you call Gate.init(). Alternatively, you can set the values programmatically using the static methods setUserConfigFile(), etc. before calling Gate.init(). Note that from version 8.5 onwards, the user config file is ignored by default unless you also call runInSandbox(false) before init. See the Javadoc documentation for details.

To set these properties when running GATE developer see the next section.

2.4 Changing GATE’s launch configuration [#]

JVM options for GATE Developer are supplied in the gate.l4j.ini file on all platforms. The gate.l4j.ini file supplied by default with GATE simply sets two standard JVM memory options:

-Xmx1G
-Xms200m

-Xmx specifies the maximum heap size in megabytes (m) or gigabytes (g), and -Xms specifies the initial size.

Note that the format consists of one option per line. All the properties listed in Section 2.3 can be configured here by prefixing them with -D, e.g., -Dgate.user.config=path/to/other-gate.xml.

Proxy configuration can be set in this file – by default GATE uses the system-wide proxy settings (-Djava.net.useSystemProxies=true) but a specific proxy can be configured by deleting that line and replacing it with settings such as:

-Dhttp.proxyHost=proxy.example.com
-Dhttp.proxyPort=8080
-Dhttp.nonProxyHosts=*.example.com

Consult the Oracle Java Networking and Proxies documentation¹ for further details of proxy configuration in Java, and see section 2.3.

For GATE Embedded, note that Java does not automatically use the system proxy settings by default, you must set java.net.useSystemProxies=true explicitly to enable this.

2.5 Configuring GATE [#]

When GATE Developer is started, or when Gate.init() is called from GATE Embedded (if you have disabled the default “sandbox” mode), GATE loads various sorts of configuration data stored as XML in a file generally called something like gate.xml or .gate.xml in your home directory. This data holds information such as:

• whether to save settings on exit;

• whether to save session on exit;

• what fonts GATE Developer should use;

• plugins to load at start;

• colours of the annotations;

• locations of files for the file chooser;

• and a lot of other GUI related options;

Configuration data can be set from the GATE Developer GUI via the ‘Options’ menu then ‘Configuration’². The user can change the appearance of the GUI in the ‘Appearance’ tab, which includes the options of font and the ‘look and feel’. The ‘Advanced’ tab enables the user to include annotation features when saving the document and preserving its format, to save the selected Options automatically on exit, and to save the session automatically on exit. These options are all stored in the user’s .gate.xml file.

2.6 Building GATE [#]

Note that you don’t need to build GATE unless you’re doing development on the system itself.

Prerequisites:

• A conforming Java environment as above.

• A clone of the relevant Git repository or repositories (see Section 2.2.3).

• A working installation of Apache Maven version 3.5.2 or newer. It is advisable that you also set your JAVA_HOME environment variable to point to the top-level directory of your Java installation.

• An appreciation of natural beauty.

To build gate-core, cd to where you cloned gate-core and:

1. Type:
   mvn install

2. [optional] To make the Javadoc documentation:
   mvn site

In order to be able to run the GATE Developer you just built, you will also need to cd into the distro folder and run mvn compile in there, in order to create the classpath file that the GATE Developer launcher uses to find the JARs.

To build plugins cd into the plugin you just cloned and run mvn install. This will build the plugin and place it in your local Maven cache, from where GATE Developer will be able to resolve it at runtime.

Note if you are building a version of a plugin that depends on a SNAPSHOT version of gate-core then you will need to add some configuration to your Maven settings.xml file, as described in the gate-core README file.

2.6.1 Using GATE with Maven/Ivy [#]

This section is based on contributions by Marin Nozhchev (Ontotext) and Benson Margulies (Basis Technology Corp).

Stable releases of GATE (since 5.2.1) are available in the standard central Maven repository, with group ID “uk.ac.gate” and artifact ID “gate-core”. To use GATE in a Maven-based project you can simply add a dependency:

<dependency>
  <groupId>uk.ac.gate</groupId>
  <artifactId>gate-core</artifactId>
  <version>8.5</version>
</dependency>

Similarly, with a project that uses Ivy for dependency management:

<dependency org="uk.ac.gate" name="gate-core" rev="8.5"/>

You do not need to do anything to allow GATE to access its plugins, it will fetch them at runtime from the internet when they are loaded.

Nightly snapshot builds of gate-core are available from our own Maven repository at http://repo.gate.ac.uk/content/groups/public.

2.7 Uninstalling GATE [#]

If you have used the installer, run:

java -jar uninstaller.jar

or just delete the whole of the installation directory (the one containing bin, lib, Uninstaller, etc.). The installer doesn’t install anything outside this directory, but for completeness you might also want to delete the settings files GATE creates in your home directory (.gate.xml and .gate.session).

2.8 Troubleshooting [#]

See the FAQ on the GATE Wiki for frequent questions about running and using GATE.

Chapter 3
Using GATE Developer [#]

This chapter introduces GATE Developer, which is the GATE graphical user interface. It is analogous to systems like Mathematica for mathematicians, or Eclipse for Java programmers, providing a convenient graphical environment for research and development of language processing software. As well as being a powerful research tool in its own right, it is also very useful in conjunction with GATE Embedded (the GATE API by which GATE functionality can be included in your own applications); for example, GATE Developer can be used to create applications that can then be embedded via the API. This chapter describes how to complete common tasks using GATE Developer. It is intended to provide a good entry point to GATE functionality, and so explanations are given assuming only basic knowledge of GATE. However, probably the best way to learn how to use GATE Developer is to use this chapter in conjunction with the demonstrations and tutorials movies. There are specific links to them throughout the chapter. There is also a complete new set of video tutorials here.

The basic business of GATE is annotating documents, and all the functionality we will introduce relates to that. Core concepts are;

• the documents to be annotated,

• corpora comprising sets of documents, grouping documents for the purpose of running uniform processes across them,

• annotations that are created on documents,

• annotation types such as ‘Name’ or ‘Date’,

• annotation sets comprising groups of annotations,

• processing resources that manipulate and create annotations on documents, and

• applications, comprising sequences of processing resources, that can be applied to a document or corpus.

What is considered to be the end result of the process varies depending on the task, but for the purposes of this chapter, output takes the form of the annotated document/corpus. Researchers might be more interested in figures demonstrating how successfully their application compares to a ‘gold standard’ annotation set; Chapter 10 in Part II will cover ways of comparing annotation sets to each other and obtaining measures such as F1. Implementers might be more interested in using the annotations programmatically; Chapter 7, also in Part II, talks about working with annotations from GATE Embedded. For the purposes of this chapter, however, we will focus only on creating the annotated documents themselves, and creating GATE applications for future use.

GATE includes a complete information extraction system that you are free to use, called ANNIE (a Nearly-New Information Extraction System). Many users find this is a good starting point for their own application, and so we will cover it in this chapter. Chapter 6 talks in a lot more detail about the inner workings of ANNIE, but we aim to get you started using ANNIE from inside of GATE Developer in this chapter.

We start the chapter with an exploration of the GATE Developer GUI, in Section 3.1. We describe how to create documents (Section 3.2) and corpora (Section 3.3). We talk about viewing and manually creating annotations (Section 3.4).

We then talk about loading the plugins that contain the processing resources you will use to construct your application, in Section 3.5. We then talk about instantiating processing resources (Section 3.7). Section 3.8 covers applications, including using ANNIE (Section 3.8.3). Saving applications and language resources (documents and corpora) is covered in Section 3.9. We conclude with a few assorted topics that might be useful to the GATE Developer user, in Section 3.11.

3.1 The GATE Developer Main Window [#]

Figure 3.1 shows the main window of GATE Developer, as you will see it when you first run it. There are five main areas:

1. at the top, the menus bar and tools bar with menus ‘File’, ‘Options’, ‘Tools’, ‘Help’ and icons for the most frequently used actions;

2. on the left side, a tree starting from ‘GATE’ and containing ‘Applications’, ‘Language Resources’ etc. – this is the resources tree;

3. in the bottom left corner, a rectangle, which is the small resource viewer;

4. in the center, containing tabs with ‘Messages’ or the name of a resource from the resources tree, the main resource viewer;

5. at the bottom, the messages bar.

The menu and the messages bar do the usual things. Longer messages are displayed in the messages tab in the main resource viewer area.

The resource tree and resource viewer areas work together to allow the system to display diverse resources in various ways. The many resources integrated with GATE can have either a small view, a large view, or both.

At any time, the main viewer can also be used to display other information, such as messages, by clicking on the appropriate tab at the top of the main window. If an error occurs in processing, the messages tab will flash red, and an additional popup error message may also occur.

In the options dialogue from the Options menu you can choose if you want to link the selection in the resources tree and the selected main view.

3.2 Loading and Viewing Documents [#]

If you right-click on ‘Language Resources’ in the resources pane, select “New’ then ‘GATE Document’, the window ‘Parameters for the new GATE Document’ will appear as shown in figure 3.2. Here, you can specify the GATE document to be created. Required parameters are indicated with a tick. The name of the document will be created for you if you do not specify it. Enter the URL of your document or use the file browser to indicate the file you wish to use for your document source. For example, you might use ‘http://gate.ac.uk’, or browse to a text or XML file you have on disk. Click on ‘OK’ and a GATE document will be created from the source you specified.

See also the movie for creating documents.

The document editor is contained in the central tabbed pane in GATE Developer. Double-click on your document in the resources pane to view the document editor.

The document editor consists of a top panel with buttons and icons that control the display of different views and the search box. Initially, you will see just the text of your document, as shown in figure 3.3. Click on ‘Annotation Sets’ and ‘Annotations List’ to view the annotation sets to the right and the annotations list at the bottom.

You will see a view similar to figure 3.4. In place of the annotations list, you can also choose to see the annotations stack. In place of the annotation sets, you can also choose to view the co-reference editor. More information about this functionality is given in Section 3.4.

Several options can be set from the small triangle icon at the top right corner.

With ‘Save Current Layout’ you store the way the different views are shown and the annotation types highlighted in the document. Then if you set ‘Restore Layout Automatically’ you will get the same views and annotation types each time you open a document. The layout is saved to the user preferences file, gate.xml. It means that you can give this file to a new user so s/he will have a preconfigured document editor.

Another setting make the document editor ‘Read-only’. If enabled, you won’t be able to edit the text but you will still be able to edit annotations. It is useful to avoid to involuntarily modify the original text.

The option ‘Right To Left Orientation’ is useful for changing orientation of the text for the languages such as Arabic and Urdu. Selecting this option changes orientation of the text of the currently visible document.

Finally you can choose between ‘Insert Append’ and ‘Insert Prepend’. That setting is only relevant when you’re inserting text at the very border of an annotation.

If you place the cursor at the start of an annotation, in one case the newly entered text will become part of the annotation, in the other case it will stay outside. If you place the cursor at the end of an annotation, the opposite will happen.

Let use this sentence: ‘This is an [annotation].’ with the square brackets [] denoting the boundaries of the annotation. If we insert a ‘x’ just before the ‘a’ or just after the ‘n’ of ‘annotation’, here’s what we get:

Append

• This is an x[annotation].

• This is an [annotationx].

Prepend

• This is an [xannotation].

• This is an [annotation]x.

Text in a loaded document can be edited in the document viewer. The usual platform specific cut, copy and paste keyboard shortcuts should also work, depending on your operating system (e.g. CTRL-C, CTRL-V for Windows). The last icon, a magnifying glass, at the top of the document editor is for searching in the document. To prevent the new annotation windows popping up when a piece of text is selected, hold down the CTRL key. Alternatively, you can hide the annotation sets view by clicking on its button at the top of the document view; this will also cause the highlighted portions of the text to become un-highlighted.

See also Section 20.2.3 for the compound document editor.

3.3 Creating and Viewing Corpora [#]

You can create a new corpus in a similar manner to creating a new document; simply right-click on ‘Language Resources’ in the resources pane, select ‘New’ then ‘GATE corpus’. A brief dialogue box will appear in which you can optionally give a name for your corpus (if you leave this blank, a corpus name will be created for you) and optionally add documents to the corpus from those already loaded into GATE.

There are three ways of adding documents to a corpus:

1. When creating the corpus, clicking on the icon next to the “documentsList” input field brings up a popup window with a list of the documents already loaded into GATE Developer. This enables the user to add any documents to the corpus.

2. Alternatively, the corpus can be loaded first, and documents added later by double clicking on the corpus and using the + and - icons to add or remove documents to the corpus. Note that the documents must have been loaded into GATE Developer before they can be added to the corpus.

3. Once loaded, the corpus can be populated by right clicking on the corpus and selecting ‘Populate’. With this method, documents do not have to have been previously loaded into GATE Developer, as they will be loaded during the population process. If you right-click on your corpus in the resources pane, you will see that you have the option to ‘Populate’ the corpus. If you select this option, you will see a dialogue box in which you can specify a directory in which GATE will search for documents. You can specify the extensions allowable; for example, XML or TXT. This will restrict the corpus population to only those documents with the extensions you wish to load. You can choose whether to recurse through the directories contained within the target directory or restrict the population to those documents contained in the top level directory. Click on ‘OK’ to populate your corpus. This option provides a quick way to create a GATE Corpus from a directory of documents.

Additionally, right-clicking on a loaded document in the tree and selecting the ‘New corpus with this document’ option creates a new transient corpus named Corpus for document name containing just this document.

See also the movie for creating and populating corpora.

Double click on your corpus in the resources pane to see the corpus editor, shown in figure 3.5. You will see a list of the documents contained within the corpus.

In the top left of the corpus editor, plus and minus buttons allow you to add documents to the corpus from those already loaded into GATE and remove documents from the corpus (note that removing a document from a corpus does not remove it from GATE).

Up and down arrows at the top of the view allow you to reorder the documents in the corpus. The rightmost button in the view opens the currently selected document in a document editor.

At the bottom, you will see that tabs entitled ‘Initialisation Parameters’ and ‘Corpus Quality Assurance’ are also available in addition to the corpus editor tab you are currently looking at. Clicking on the ‘Initialisation Parameters’ tab allows you to view the initialisation parameters for the corpus. The ‘Corpus Quality Assurance’ tab allows you to calculate agreement measures between the annotations in your corpus. Agreement measures are discussed in depth in Chapter 10. The use of corpus quality assurance is discussed in Section 10.3.

3.4 Working with Annotations [#]

In this section, we will talk in more detail about viewing annotations, as well as creating and editing them manually. As discussed in at the start of the chapter, the main purpose of GATE is annotating documents. Whilst applications can be used to annotate the documents entirely automatically, annotation can also be done manually, e.g. by the user, or semi-automatically, by running an application over the corpus and then correcting/adding new annotations manually. Section 3.4.5 focuses on manual annotation. In Section 3.7 we talk about running processing resources on our documents. We begin by outlining the functionality around viewing annotations, organised by the GUI area to which the functionality pertains.

3.4.1 The Annotation Sets View [#]

To view the annotation sets, click on the ‘Annotation Sets’ button at the top of the document editor, or use the F3 key (see Section 3.10 for more keyboard shortcuts). This will bring up the annotation sets viewer, which displays the annotation sets available and their corresponding annotation types.

The annotation sets view is displayed on the left part of the document editor. It’s a tree-like view with a root for each annotation set. The first annotation set in the list is always a nameless set. This is the default annotation set. You can see in figure 3.4 that there is a drop-down arrow with no name beside it. Other annotation sets on the document shown in figure 3.4 are ‘Key’ and ‘Original markups’. Because the document is an XML document, the original XML markup is retained in the form of an annotation set. This annotation set is expanded, and you can see that there are annotations for ‘TEXT’, ‘body’, ‘font’, ‘html’, ‘p’, ‘table’, ‘td’ and ‘tr’.

To display all the annotations of one type, tick its checkbox or use the space key. The text segments corresponding to these annotations will be highlighted in the main text window. To delete an annotation type, use the delete key. To change the color, use the enter key. There is a context menu for all these actions that you can display by right-clicking on one annotation type, a selection or an annotation set.

If you keep shift key pressed when you open the annotation sets view, GATE Developer will try to select any annotations that were selected in the previous document viewed (if any); otherwise no annotation will be selected.

Having selected an annotation type in the annotation sets view, hovering over an annotation in the main resource viewer or right-clicking on it will bring up a popup box containing a list of the annotations associated with it, from which one can select an annotation to view in the annotation editor, or if there is only one, the annotation editor for that annotation. Figure 3.6 shows the annotation editor.

3.4.2 The Annotations List View [#]

To view the list of annotations and their features, click on the ‘Annotations list’ button at the top of the main window or use F4 key. The annotation list view will appear below the main text. It will only contain the annotations selected from the annotation sets view. These lists can be sorted in ascending and descending order for any column, by clicking on the corresponding column heading. Moreover you can hide a column by using the context menu by right-clicking on the column headings. Selecting rows in the table will blink the respective annotations in the document. Right-click on a row or selection in this view to delete or edit an annotation. Delete key is a shortcut to delete selected annotations.

3.4.3 The Annotations Stack View [#]

This view is similar to the ANNIC view described in section 9.2. It displays annotations at the document caret position with some context before and after. The annotations are stacked from top to bottom, which gives a clear view when they are overlapping.

As the view is centred on the document caret, you can use the conventional key to move it and update the view: notably the keys left and right to skip one letter; control + left/right to skip one word; up and down to go one line up or down; and use the document scrollbar then click in the document to move further.

There are two buttons at the top of the view that centre the view on the closest previous/next annotation boundary among all displayed. This is useful when you want to skip a region without annotation or when you want to reach the beginning or end of a very long annotation.

The annotation types displayed correspond to those selected in the annotation sets view. You can display feature values for an annotation rectangle by hovering the mouse on it or select only one feature to display by double-clicking on the annotation type in the first column.

Right-click on an annotation in the annotations stack view to edit it. Control-Shift-click to delete it. Double-click to copy it to another annotation set. Control-click on a feature value that contains an URL to display it in your browser.

All of these mouse shortcuts make it easier to create a gold standard annotation set.

3.4.4 The Co-reference Editor [#]

The co-reference editor allows co-reference chains (see Section 6.9) to be displayed and edited in GATE Developer. To display the co-reference editor, first open a document in GATE Developer, and then click on the Co-reference Editor button in the document viewer.

The combo box at the top of the co-reference editor allows you to choose which annotation set to display co-references for. If an annotation set contains no co-reference data, then the tree below the combo box will just show ‘Coreference Data’ and the name of the annotation set. However, when co-reference data does exist, a list of all the co-reference chains that are based on annotations in the currently selected set is displayed. The name of each co-reference chain in this list is the same as the text of whichever element in the chain is the longest. It is possible to highlight all the member annotations of any chain by selecting it in the list.

When a co-reference chain is selected, if the mouse is placed over one of its member annotations, then a pop-up box appears, giving the user the option of deleting the item from the chain. If the only item in a chain is deleted, then the chain itself will cease to exist, and it will be removed from the list of chains. If the name of the chain was derived from the item that was deleted, then the chain will be given a new name based on the next longest item in the chain.

A combo box near the top of the co-reference editor allows the user to select an annotation type from the current set. When the Show button is selected all the annotations of the selected type will be highlighted. Now when the mouse pointer is placed over one of those annotations, a pop-up box will appear giving the user the option of adding the annotation to a co-reference chain. The annotation can be added to an existing chain by typing the name of the chain (as shown in the list on the right) in the pop-up box. Alternatively, if the user presses the down cursor key, a list of all the existing annotations appears, together with the option [New Chain]. Selecting the [New Chain] option will cause a new chain to be created containing the selected annotation as its only element.

Each annotation can only be added to a single chain, but annotations of different types can be added to the same chain, and the same text can appear in more than one chain if it is referenced by two or more annotations.

The movie for inspecting results is also useful for learning about viewing annotations.

3.4.5 Creating and Editing Annotations [#]

To create annotations manually, select the text you want to annotate and hover the mouse on the selection or use control+E keys. A popup will appear, allowing you to create an annotation, as shown in figure 3.9

The type of the annotation, by default, will be the same as the last annotation you created, unless there is none, in which case it will be ‘_New_’. You can enter any annotation type name you wish in the text box, unless you are using schema-driven annotation (see Section 3.4.6). You can add or change features and their values in the table below.

To delete an annotation, click on the red X icon at the top of the popup window. To grow/shrink the span of the annotation at its start use the two arrow icons on the left or right and left keys. Use the two arrow icons next on the right to change the annotation end or alt+right and alt+left keys. Add shift and control+shift keys to make the span increment bigger. The red X icon is for removing the annotation.

The pin icon is to pin the window so that it remains where it is. If you drag and drop the window, this automatically pins it too. Pinning it means that even if you select another annotation (by hovering over it in the main resource viewer) it will still stay in the same position.

The popup menu only contains annotation types present in the Annotation Schema and those already listed in the relevant Annotation Set. To create a new Annotation Schema, see Section 3.4.6. The popup menu can be edited to add a new annotation type, however.

The new annotation created will automatically be placed in the annotation set that has been selected (highlighted) by the user. To create a new annotation set, type the name of the new set to be created in the box below the list of annotation sets, and click on ‘New’.

Figure 3.10 demonstrates adding a ‘Organization’ annotation for the string ‘EPSRC’ (highlighted in green) to the default annotation set (blank name in the annotation set view on the right) and a feature name ‘type’ with a value about to be added.

To add a second annotation to a selected piece of text, or to add an overlapping annotation to an existing one, press the CTRL key to avoid the existing annotation popup appearing, and then select the text and create the new annotation. Again by default the last annotation type to have been used will be displayed; change this to the new annotation type. When a piece of text has more than one annotation associated with it, on mouseover all the annotations will be displayed. Selecting one of them will bring up the relevant annotation popup.

To search and annotate the document automatically, use the search and annotate function as shown in figure 3.11:

• Create and/or select an annotation to be used as a model to annotate.

• Open the panel at the bottom of the annotation editor window.

• Change the expression to search if necessary.

• Use the [First] button or Enter key to select the first expression to annotate.

• Use the [Annotate] button if the selection is correct otherwise the [Next] button. After a few cycles of [Annotate] and [Next], Use the [Ann. all next] button.

Note that after using the [First] button you can move the caret in the document and use the [Next] button to avoid continuing the search from the beginning of the document. The [?] button at the end of the search text field will help you to build powerful regular expressions to search.

3.4.6 Schema-Driven Editing [#]

Annotation schemas allow annotation types and features to be pre-specified, so that during manual annotation, the relevant options appear on the drop-down lists in the annotation editor. You can see some example annotation schemas in Section 5.4.1. Annotation schemas provide a means to define types of annotations in GATE Developer. Basically this means that GATE Developer ‘knows about’ annotations defined in a schema. Annotation schemas are supported by the ‘Annotation schema’ language resource, which is one of the default LR types (along with corpus and document) available in GATE without the need to load any plugins.

To load an annotation schema into GATE Developer, right-click on ‘Language Resources’ in the resources pane. Select ‘New’ then ‘Annotation schema’. A popup box will appear in which you can browse to your annotation schema XML file. A default set of annotation schemas for common annotation types including Person, Organization and Location is provided in the ANNIE plugin, and can be loaded by creating an Annotation schema LR from the file plugins/ANNIE/resources/schema/ANNIE-Schemas.xml in the GATE distribution. You can also define your own schemas to tell GATE Developer about other kinds of annotations you frequently use. Each schema file can define only one annotation type, but you can have a master file which includes others, in order to load a group of schemas in one operation. The ANNIE schemas provide an example of this technique.

By default GATE Developer will allow you to create any annotations in a document, whether or not there is a schema to describe them. An alternative annotation editor component is available which constrains the available annotation types and features much more tightly, based on the annotation schemas that are currently loaded. This is particularly useful when annotating large quantities of data or for use by less skilled users.

To use this, you must load the Schema_Annotation_Editor plugin. With this plugin loaded, the annotation editor will only offer the annotation types permitted by the currently loaded set of schemas, and when you select an annotation type only the features permitted by the schema are available to edit¹. Where a feature is declared as having an enumerated type the available enumeration values are presented as an array of buttons, making it easy to select the required value quickly.

3.4.7 Printing Text with Annotations [#]

We suggest you to use your browser to print a document as GATE don’t propose a printing facility for the moment.

First save your document by right clicking on the document in the left resources tree then choose ‘Save Preserving Format’. You will get an XML file with all the annotations highlighted as XML tags plus the ‘Original markups’ annotations set.

It’s possible that the output will not have an XML header and footer because the document was created from a plain text document. In that case you can use the XHTML example below.

Then add a stylesheet processing instruction at the beginning of the XML file, the second line in the following minimalist XHTML document:

<?xml version="1.0" encoding="UTF-8" ?>
<?xml-stylesheet type="text/css" href="gate.css"?>
<!DOCTYPE html
     PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <title>Virtual Library</title>
  </head>
  <body>
  <p>Content of the document</p>
  ...
  </body>
</html>

And create a file ‘gate.css’ in the same directory:

BODY, body { margin: 2em } /* or any other first level tag */
P, p { display: block } /* or any other paragraph tag */
/* ANNIE tags but you can use whatever tags you want */
/* be careful that XML tags are case sensitive */
Date         { background-color: rgb(230, 150, 150) }
FirstPerson  { background-color: rgb(150, 230, 150) }
Identifier   { background-color: rgb(150, 150, 230) }
JobTitle     { background-color: rgb(150, 230, 230) }
Location     { background-color: rgb(230, 150, 230) }
Money        { background-color: rgb(230, 230, 150) }
Organization { background-color: rgb(230, 200, 200) }
Percent      { background-color: rgb(200, 230, 200) }
Person       { background-color: rgb(200, 200, 230) }
Title        { background-color: rgb(200, 230, 230) }
Unknown      { background-color: rgb(230, 200, 230) }
Etc          { background-color: rgb(230, 230, 200) }
/* The next block is an example for having a small tag
   with the name of the annotation type after each annotation */
Date:after {
content: "Date";
font-size: 50%;
vertical-align: sub;
color: rgb(100, 100, 100);
}

Finally open the XML file in your browser and print it.

Note that overlapping annotations, cannot be expressed correctly with inline XML tags and thus won’t be displayed correctly.

3.5 Using CREOLE Plugins [#]

In GATE, processing resources are used to automatically create and manipulate annotations on documents. We will talk about processing resources in the next section. However, we must first introduce CREOLE plugins. In most cases, in order to use a particular processing resource (and certain language resources) you must first load the CREOLE plugin that contains it. This section talks about using CREOLE plugins. Then, in Section 3.7, we will talk about creating and using processing resources.

The definitions of CREOLE resources (e.g. processing resources such as taggers and parsers, see Chapter 4) are stored in Maven central repository.

Plugins can have one or more of the following states in relation with GATE:

known

plugins are those plugins that the system knows about. These include all the plugins: 1. default plugins provided by Gate team. 2. The plugins added by the user manually according to the Maven artifact id. 3. those installed in the user’s own plugin directory.

loaded

plugins are the plugins currently loaded in the system. All CREOLE resource types from the loaded plugins are available for use. All known plugins can easily be loaded and unloaded using the user interface.

auto-loadable

plugins are the list of plugins that the system loads automatically during initialisation which can be configured via the load.plugin.path system property.

As hinted at above plugins can be loaded from numerous sources:

core plugins

are distributed with GATE to the Maven central repository.

maven plugins

are distributed with other parties to the Maven central repository.

user plugins

are plugins that have been installed by the user into their personal plugins folder. The location of this folder can be set either through the configuration tab of the CREOLE manager interface or via the gate.user.plugins system property

remote plugins

are plugins which are loaded via http from a remote machine.

Regular Maven users may have additional repositories or “mirror” settings configured in their m2/settings.xml file – GATE will respect these settings when retrieving plugins, including authentication with encrypted passwords in the standard Maven way with a master password in .m2/settings-security.xml. In particular if you have an organisational Maven repository configured as a <mirrorOf>external:*</mirrorOf> then this will be accepted and GATE will not attempt to use the Central Repository directly.

The CREOLE plugins can be managed through the graphical user interface which can be activated by selecting ‘Manage CREOLE Plugins’ from the ‘File’ menu. This will bring up a window listing all the known plugins. For each plugin there are two check-boxes – one labelled ‘Load Now’, which will load the plugin, and the other labelled ‘Load Always’ which will add the plugin to the list of auto-loadable plugins. A ‘Delete’ button is also provided – which will remove the plugin from the list of known plugins. This operation does not delete the actual plugin directory. Installed plugins are found automatically when GATE is started; if an installed plugin is deleted from the list, it will re-appear next time GATE is launched.

If you select a plugin, you will see in the pane on the right the list of resources that plugin contains. For example, in figure 3.12, the ‘ANNIE’ plugin is selected, and you can see that it contains 17 resources. If you wish to use a particular resource you will have to ascertain which plugin contains it. This list can be useful for that.

Having loaded the plugins you need, the resources they define will be available for use. Typically, to the GATE Developer user, this means that they will appear on the ‘New’ menu when you right-click on ‘Processing Resources’ in the resources pane, although some special plugins have different effects; for example, the Schema_Annotation_Editor (see Section 3.4.6).

Some plugins also contain files which are used to configure the resources. For example, the ANNIE plugin contains the resources for the ANNIE Gazetteer and the ANNIE NE Transducer (amongst other things). While often these files can be used straight from within the plugin, it can be useful to edit them, either to add missing information or as a starting point for delveloping new resources etc. To extract a copy of these resource files from a plugin simply select it in the plugin manager and then click the download resources button shown under the list of resources the plugin defines. This button will only be enabled for plugins which contain such files. After clicking the button you will be asked to select a directory into which to copy the files. You can then edit the files as needed before using them to configure a new instance of the appropriate processing resource.

3.6 Installing and updating CREOLE Plugins [#]

While GATE is distributed with a number of core plugins (see Part III) there are many more plugins developed and made available by other GATE users. Some of these additional plugins can easily be installed into your local copy of GATE through the CREOLE plugin manager.

Installing new plugins is simply a case of checking the box and clicking ‘Apply All’. Note that plugins are installed into the user plugins directory, which must have been correctly configured before you can try installing new plugins.

Once a plugin is installed it will appear in the list of ‘Installed Plugins’ and can be loaded in the same way as any other CREOLE plugin (see Section 3.7). If a new version of a plugin you have installed becomes available the new version will be offered as an update. These updates can be installed in the same way as a new plugin.

To register a new plugin just need simply click the ‘+’ button located at the top right corner of Plugin Manager Then you can either register a new plugin by provide the Maven Group and Artifact ID for maven plugins or provide the Dirctory URL for local or remote plugins.

3.7 Loading and Using Processing Resources [#]

This section describes how to load and run CREOLE resources not present in ANNIE. To load ANNIE, see Section 3.8.3. For technical descriptions of these resources, see the appropriate chapter in Part III (e.g. Chapter 23). First ensure that the necessary plugins have been loaded (see Section 3.5). If the resource you require does not appear in the list of Processing Resources, then you probably do not have the necessary plugin loaded. Processing resources are loaded by selecting them from the set of Processing Resources: right click on Processing Resources or select ‘New Processing Resource’ from the File menu.

For example, use the Plugin Console Manager to load the ‘Tools’ plugin. When you right click on ‘Processing Resources’ in the resources pane and select ‘New’ you have the option to create any of the processing resources that plugin provides. You may choose to create a ‘GATE Morphological Analyser’, with the default parameters. Having done this, an instance of the GATE Morphological Analyser appears under ‘Processing Resources’. This processing resource, or PR, is now available to use. Double-clicking on it in the resources pane reveals its initialisation parameters, see figure 3.14.

This processing resource is now available to be added to applications. It must be added to an application before it can be applied to documents. You may create as many of a particular processing resource as you wish, for example with different initialisation parameters. Section 3.8 talks about creating and running applications.

See also the movie for loading processing resources.

3.8 Creating and Running an Application [#]

Once all the resources you need have been loaded, an application can be created from them, and run on your corpus. Right click on ‘Applications’ and select ‘New’ and then either ‘Corpus Pipeline’ or ‘Pipeline’. A pipeline application can only be run over a single document, while a corpus pipeline can be run over a whole corpus.

To build the pipeline, double click on it, and select the resources needed to run the application (you may not necessarily wish to use all those which have been loaded).

Transfer the necessary components from the set of ‘loaded components’ displayed on the left hand side of the main window to the set of ‘selected components’ on the right, by selecting each component and clicking on the left and right arrows, or by double-clicking on each component.

Ensure that the components selected are listed in the correct order for processing (starting from the top). If not, select a component and move it up or down the list using the up/down arrows at the left side of the pane.

Ensure that any parameters necessary are set for each processing resource (by clicking on the resource from the list of selected resources and checking the relevant parameters from the pane below). For example, if you wish to use annotation sets other than the Default one, these must be defined for each processing resource.

Note that if a corpus pipeline is used, the corpus needs only to be set once, using the drop-down menu beside the ‘corpus’ box. If a pipeline is used, the document must be selected for each processing resource used.

Finally, click on ‘Run’ to run the application on the document or corpus.

See also the movie for loading and running processing resources.

For how to use the conditional versions of the pipelines see Section 3.8.2 and for saving/restoring the configuration of an application see Section 3.9.3.

3.8.1 Running an Application on a Datastore [#]

To avoid loading all your documents at the same time you can run an application on a datastore corpus.

To do this you need to load your datastore, see section 3.9.2, and to load the corpus from the datastore by double clicking on it in the datastore viewer.

Then, in the application viewer, you need to select this corpus in the drop down list of corpora.

When you run the application on the corpus datastore, each document will be loaded, processed, saved then unloaded. So at any time there will be only one document from the datastore corpus loaded. This prevent memory shortage but is also a little bit slower than if all your documents were already loaded.

The processed documents are automatically saved back to the datastore so you may want to use a copy of the datastore to experiment.

Be very careful that if you have some documents from the datastore corpus already loaded before running the application then they will not be unloaded nor saved. To save such document you have to right click on it in the resources tree view and save it to the datastore.

3.8.2 Running PRs Conditionally on Document Features [#]

The ‘Conditional Pipeline’ and ‘Conditional Corpus Pipeline’ application types are conditional versions of the pipelines mentioned in Section 3.8 and allow processing resources to be run or not according to the value of a feature on the document. In terms of graphical interface, the only addition brought by the conditional versions of the applications is a box situated underneath the lists of available and selected resources which allows the user to choose whether the currently selected processing resource will run always, never or only on the documents that have a particular value for a named feature.

If the Yes option is selected then the corresponding resource will be run on all the documents processed by the application as in the case of non-conditional applications. If the No option is selected then the corresponding resource will never be run; the application will simply ignore its presence. This option can be used to temporarily and quickly disable an application component, for debugging purposes for example.

The If value of feature option permits running specific application components conditionally on document features. When selected, this option enables two text input fields that are used to enter the name of a feature and the value of that feature for which the corresponding processing resource will be run. When a conditional application is run over a document, for each component that has an associated condition, the value of the named feature is checked on the document and the component will only be used if the value entered by the user matches the one contained in the document features.

At first sight the conditional behaviour available with these controller may seem limited, but in fact it is very powerful when used in conjunction with JAPE grammars (see chapter 8). Complex conditions can be encoded in JAPE rules which set the appropriate feature values on the document for use by the conditional controllers. Alternatively, the Groovy plugin provides a scriptable controller (see section 7.16.3) in which the execution strategy is defined by a Groovy script, allowing much richer conditional behaviour to be encoded directly in the controller’s configuration.

3.8.3 Doing Information Extraction with ANNIE [#]

This section describes how to load and run ANNIE (see Chapter 6) from GATE Developer. ANNIE is a good place to start because it provides a complete information extraction application, that you can run on any corpus. You can then view the effects.

From the File menu, select ‘Load ANNIE System’. To run it in its default state, choose ‘with Defaults’. This will automatically load all the ANNIE resources, and create a corpus pipeline called ANNIE with the correct resources selected in the right order, and the default input and output annotation sets.

If ‘without Defaults’ is selected, the same processing resources will be loaded, but a popup window will appear for each resource, which enables the user to specify a name, location and other parameters for the resource. This is exactly the same procedure as for loading a processing resource individually, the difference being that the system automatically selects those resources contained within ANNIE. When the resources have been loaded, a corpus pipeline called ANNIE will be created as before.

The next step is to add a corpus (see Section 3.3), and select this corpus from the drop-down corpus menu in the Serial Application editor. Finally click on ‘Run’ from the Serial Application editor, or by right clicking on the application name in the resources pane and selecting ‘Run’. (Many people prefer to switch to the messages tab, then run their application by right-clicking on it in the resources pane, because then it is possible to monitor any messages that appear whilst the application is running.)

To view the results, double click on one of the document contained in the corpus processed in the left hand tree view. No annotation sets nor annotations will be shown until annotations are selected in the annotation sets; the ‘Default’ set is indicated only with an unlabelled right-arrowhead which must be selected in order to make visible the available annotations. Open the default annotation set and select some of the annotations to see what the ANNIE application has done.

See also the movie for loading and running ANNIE.

3.8.4 Modifying ANNIE [#]

You will need to first make a copy of ANNIE resources by extracting them from the ANNIE plugin via the plugin manager. Once you have a copy of the resources simply locate the file(s) you want to modify, edit them, and then use them to configure the appropriate ANNIE processing resources.

3.9 Saving Applications and Language Resources [#]

In this section, we will describe how applications and language resources can be saved for use outside of GATE and for use with GATE at a later time. Section 3.9.1 talks about saving documents to file. Section 3.9.2 outlines how to use datastores. Section 3.9.3 talks about saving application states (resource parameter states), and Section 3.9.4 talks about exporting applications together with referenced files and resources to a ZIP file.

3.9.1 Saving Documents to File [#]

There are three main ways to save annotated documents:

1. in GATE’s own XML serialisation format (including all the annotations on the document);

2. an inline XML format that saves the original markup and selected annotations

3. by writing your own exporter algorithm as a processing resource

This section describes how to use the first two options.

Both types of data export are available in the popup menu triggered by right-clicking on a document in the resources tree (see Section 3.1) and selecting the “Save As...” menu. In addition, all documents in a corpus can be saved as individual XML files into a directory by right-clicking on the corpus instead of individual documents.

Selecting to save as GATE XML leads to a file open dialogue; give the name of the file you want to create, and the whole document and all its data will be exported to that file. If you later create a document from that file, the state will be restored. (Note: because GATE’s annotation model is richer than that of XML, and because our XML dump implementation sometimes cuts corners², the state may not be identical after restoration. If your intention is to store the state for later use, use a DataStore instead.)

The ‘Inline XML’ option leads to a richer dialog than the ‘GATE XML’ option. This allows you to select the file to save to at the top of the dialog box, but also then allows you to configure extactly what is saved and how. By default the exporter is configured to save all the annotations from ‘Original markups’ (i.e. those extracted from the source document when it was loaded) as well as Person, Organization, and Location from the default set (i.e. the main output annotations from running ANNIE). Features of these annotations will also be saved.

The annotations are saved as normal XML document tags, using the annotation type as the tag name. If you choose to save features then they will be added as attributes to the relevant XML tags.

Note that GATE’s model of annotation allows graph structures, which are difficult to represent in XML (XML is a tree-structured representation format). During the dump process, annotations that cross each other in ways that cannot be represented in legal XML will be discarded, and a warning message printed.

Saving documents using this ‘Inine XML’ format that were not created from an HTML or XML file often results in a plain text file, with in-line tags for the saved annotations. In otherwords, if the set of annotations you are saving does not include an annotation which spans the entire document, the result will not be valid XML and may not load back into GATE. This format should really be considered a legacy format, and it may be removed in future versions of GATE.

3.9.2 Saving and Restoring LRs in Datastores [#]

Where corpora are large, the memory available may not be sufficient to have all documents open simultaneously. The datastore functionality provides the option to save documents to disk and open them only one at a time for processing. This means that much larger corpora can be used. A datastore can also be useful for saving documents in an efficient and lossless way.

To save a text in a datastore, a new datastore must first be created if one does not already exist. Create a datastore by right clicking on Datastore in the left hand pane, and select the option ‘Create Datastore’. Select the data store type you wish to use. Create a directory to be used as the datastore (note that the datastore is a directory and not a file).

You can either save a whole corpus to the datastore (in which case the structure of the corpus will be preserved) or you can save individual documents. The recommended method is to save the whole corpus. To save a corpus, right click on the corpus name and select the ‘Save to...’ option (giving the name of the datastore created earlier). To save individual documents to the datastore, right clicking on each document name and follow the same procedure.

To load a document from a datastore, do not try to load it as a language resource. Instead, open the datastore by right clicking on Datastore in the left hand pane, select ‘Open Datastore’ and choose the datastore to open. The datastore tree will appear in the main window. Double click on a corpus or document in this tree to open it. To save a corpus and document back to the same datastore, simply select the ‘Save’ option.

See also the movie for creating a datastore and the movie for loading corpus and documents from a datastore.

3.9.3 Saving Application States to a File [#]

Resources, and applications that are made up of them, are created based on the settings of their parameters (see Section 3.7). It is possible to save the data used to create an application to a file and re-load it later. To save the application to a file, right click on it in the resources tree and select ‘Save application state’, which will give you a file creation dialogue. Choose a file name that ends in gapp as this file dialog and the one for loading application states age displays all files which have a name ending in gapp. A common convention is to use .gapp or .xgapp as a file extension.

To restore the application later, select ‘Restore application from file’ from the ‘File’ menu.

Note that the data that is saved represents how to recreate an application – not the resources that make up the application itself. So, for example, if your application has a resource that initialises itself from some file (e.g. a grammar, a document) then that file must still exist when you restore the application.

In case you don’t want to save the corpus configuration associated with the application then you must select ‘<none>’ in the corpus list of the application before saving the application.

The file resulting from saving the application state contains the values of the initialisation and runtime parameters for all the processing resources contained by the stored application as well as the values of the initialisation parameters for all the language resources referenced by those processing resources. Note that if you reference a document that has been created with an empty URL and empty string content parameter and subsequently been manually edited to add content, that content will not be saved. In order for document content to be preserved, load the document from an URL, specify the content as for the string content parameter or use a document from a datastore.

For the parameters of type URL or “ResourceReference” (which are typically used to select resources such as grammars or rules files either from inside the plugin or elsewhere on disk) a transformation is applied so that the paths are are stored relative to either the location of the saved application state file or a special user resources home directory, according to the following rules:

• If the property gate.user.resourceshome is set to the path of a directory and the resource is located inside that directory but the state file is saved to a location outside of this directory, the path is stored relative to this directory and the path marker $resourceshome$ is used.

• in all other situations, the path is stored relative to the location of the application state file location and the the path marker $relpath$ is used.

References to resources inside GATE plugins are stored as a special type of URI of the form creole://group;artifact;version/path/inside/plugin. In this way, all resource files that are part of plugins are always used corretly, no matter where the plugins are stored. Resource files which are not part of a plugin and used by an application do not need to be in the same location as when the application was initially created but rather in the same location relative to the location of the application file. In addition if your application uses a project-specific location for global resources or project specific plugins, the java property gate.user.resourceshome can be set to this location and the application will be stored so that this location will also always be used correctly, no matter where the application state file is copied to. To set the resources home directory, the -rh location option for the Linux script gate.sh to start GATE can be used. The combination of these features allows the creation and deployment of portable applications by keeping the application file and the resource files used by the application together.

If your application uses resources from inside plugins then those resources may change if you upgrade your application to a newer version of the plugin. If you want to upgrade to a newer plugin but keep the same resources you should export a copy of the resource files from the plugin onto disk and load them from there instead of using the plugin-relative defaults.

When an application is restored from an application state file, GATE uses the keyword $relpath$ for paths relative to the location of the gapp file and $resourceshom$ for paths relative to the the location the property gate.user.resourceshome is set. There exists other keywords that can be interesting in some cases. You will need to edit the gapp file manually. You can use $sysprop:...$ to declare paths relative to any java system property, for example $sysprop:user.home$.

If you want to save your application along with all plugins and resources it requires you can use the ‘Export for GATE Cloud’ option (see Section 3.9.4).

See also the movie for saving and restoring applications.

3.9.4 Saving an Application with its Resources (e.g. GATE Cloud) [#]

When you save an application using the ‘Save application state’ option (see Section 3.9.3), the saved file contains references to the plugins that were loaded when the application was saved, and to any resource files required by the application. To be able to reload the file, these plugins and other dependencies must exist at the same locations (relative to the saved state file). While this is fine for saving and loading applications on a single machine it means that if you want to package your application to run it elsewhere (e.g. deploy it to GATE Cloud) then you need to be careful to include all the resource files and plugins at the right locations in your package. The ‘Export for GATE Cloud’ option on the right-click menu for an application helps to automate this process.

When you export an application in this way, GATE Developer produces a ZIP file containing the saved application state (in the same format as ‘Save application state’). Any plugins and resource files that the application refers to are also included in the zip file, and the relative paths in the saved state are rewritten to point to the correct locations within the package. The resulting package is therefore self-contained and can be copied to another machine and unpacked there, or passed to GATE Cloud for deployment. Maven-style plugins will be resolved from within the package rather than being downloaded at runtime from the internet.

There are a few important points to note about the export process:

• All plugins that are loaded at the point when you perform the export will be included in the resulting package. Use the plugin manager to unload any plugins your application is not using before you export it.

• If your application refers to a resource file that is in a directory on disk rather than inside one of the loaded plugins, the entire contents of this directory will be recursively included in the package. If you have a number of unrelated resources in a single directory (e.g. many sets of large gazetteer lists) you may want to separate them into separate directories so that only the relevant ones are included in the package.

• The packager only knows about resources that your application refers to directly in its parameters. For example, if your application includes a multi-phase JAPE grammar the packager will only consider the main grammar file, not any of its sub-phases. If the sub-phases are not contained in the same directory as the main grammar you may find they are not included. If indirect references of this kind are all to files under the same directory as the ‘master’ file it will work OK.

If you require more flexibility than this option provides you should read Section E.2, which describes the underlying Ant task that the exporter uses.

3.9.5 Upgrade An Application to use Newer Versions of Plugins [#]

Some of the changes introduced in GATE 8.5 mean that applications saved with a previous version of GATE might not load without being updated. Loading such an application is likely to result in errors similar to those seen in Figure 3.15.

In order to load such application into GATE 8.5 (or above), you need first upgrade them to use compatible versions of the relevant plugins. In most cases this process can be automated and we provide a tool to walk you through the process. To start upgrading an application select ‘Upgrade XGapp’ from the ‘Tools’ menu. This will first ask you to choose an application file to upgrade and will then present the UI shown in Figure 3.16.

One the application has been analysed the tool will show you a table in which each row signifies a plugin used by the app. In the left most column it lists the plugin currently referenced by the application. This is followed by details of the new plugin. While in most cases the tool can correctly determine the right plugin to offer in this column you can correct any mistakes by double-clicking the incorrect plugin and then specifying the correct plugin location. The final two columns determine if the plugin is upgraded and to which version. The versions offered are all those which are available and known to be compatible with the version of GATE you are running. By default the latest available version will be selected, although -SNAPSHOT versions are only selected by default if you are also running a -SNAPSHOT version of GATE.

The ‘Upgrade’ column allows you to determine if and how a plugin will be upgraded. The three possible choices are Upgrade, Plugin Only, and Skip. Skip is fairly self explanatory but upgrade and plugin only require a little more explanation. Upgrade means that not only will the plugin location be upgraded, but also any resources that reside within the plugin will also be changed to reference those within the new plugin. This is the only upgrade option when considering a plugin which was originally part of the GATE distribution. The plugin only option allows you to change the application to load a new version of the plugin which leaving the resource locations untouched. This is useful for cases where you have edited the resources inside a plugin rather than having created a separate copy specific to the application.

After upgrade, the old version of the application file will still be available but will have been renamed by adding the ’.bak’ suffix.

In most cases this upgrade process will work without issue. If, however, you find you have an application which fails to open after the upgrade then it maybe because one or more plugins couldn’t be correctly mapped to new versions. In these cases the best option is to revert the upgrade (replace the xgapp file with the generated backup), load the application into GATE 8.4.1 and then use the “Export for GATE Cloud” option to produce a self contained application (see Section 3.9.4). Then finally run the upgrade tool over this version of the application.

The two buttons at the top of the dialog allow you save and restore the mappings defined in the table. This makes it easier to upgrade a set of related applications which should all be upgraded in a similar fashion.

Note that this process is not limited simply to upgrading applications saved prior to GATE 8.5 but can be used at any time to upgrade the version of a plugin used by an application.

3.10 Keyboard Shortcuts [#]

You can use various keyboard shortcuts for common tasks in GATE Developer. These are listed in this section.

General (Section 3.1):

• F1 Display a help page for the selected component

• Alt+F4 Exit the application without confirmation

• Tab Put the focus on the next component or frame

• Shift+Tab Put the focus on the previous component or frame

• F6 Put the focus on the next frame

• Shift+F6 Put the focus on the previous frame

• Alt+F Show the File menu

• Alt+O Show the Options menu

• Alt+T Show the Tools menu

• Alt+H Show the Help menu

• F10 Show the first menu

Resources tree (Section 3.1):

• Enter Show the selected resources

• Ctrl+H Hide the selected resource

• Ctrl+Shift+H Hide all the resources

• F2 Rename the selected resource

• Ctrl+F4 Close the selected resource

Document editor (Section 3.2):

• Ctrl+F Show the search dialog for the document

• Ctrl+E Edit the annotation at the caret position

• Ctrl+S Save the document in a file

• F3 Show/Hide the annotation sets

• Shift+F3 Show the annotation sets with preselection

• F4 Show/Hide the annotations list

• F5 Show/Hide the coreference editor

• F7 Show/Hide the text

Annotation editor (Section 3.4):

• Right/Left Grow/Shrink the annotation span at its start

• Alt+Right/Alt+Left Grow/Shrink the annotation span at its end

• +Shift/+Ctrl+Shift Use a span increment of 5/10 characters

• Alt+Delete Delete the currently edited annotation

Annic/Lucene datastore (Chapter 9):

• Alt+Enter Search the expression in the datastore

• Alt+Backspace Delete the search expression

• Alt+Right Display the next page of results

• Alt+Left Display the row manager

• Alt+E Export the results to a file

Annic/Lucene query text field (Chapter 9):

• Ctrl+Enter Insert a new line

• Enter Search the expression

• Alt+Top Select the previous result

• Alt+Bottom Select the next result

3.11 Miscellaneous [#]

3.11.1 Stopping GATE from Restoring Developer Sessions/Options [#]

GATE can remember Developer options and the state of the resource tree when it exits. The options are saved by default; the session state is not saved by default. This default behaviour can be changed from the ‘Advanced’ tab of the ‘Configuration’ choice on the ‘Options’ menu (or the ‘Preferences’ option on the ‘GATE’ application menu on Mac).

If a problem occurs and the saved data prevents GATE Developer from starting, you can fix this by deleting the configuration and session data files. These are stored in your home directory, and are called gate.xml and gate.sesssion or .gate.xml and .gate.sesssion depending on platform. On Windows your home is typically:

95, 98, NT:

Windows Directory/profiles/username

2000, XP:

Windows Drive/Documents and Settings/username

Windows 7 or later

Windows Drive/Users/username

though the directory name may be in your local language if your copy of Windows is not in English.

3.11.2 Working with Unicode [#]

When you create a document from a URL pointing to textual data in GATE, you have to tell the system what character encoding the text is stored in. By default, GATE will set this parameter to be the empty string. This tells Java to use the default encoding for whatever platform it is running on at the time – e.g. on Western versions of Windows this will be ISO-8859-1, and Eastern ones ISO-8859-9. On Linux systems, the default encoding is influenced by the LANG environment variable, e.g. when this variable is set to en_US.utf-8 the default encoding used will be UTF-8. You can change the default encoding used by GATE to UTF-8 by adding -Dfile.encoding=UTF-8 to the gate.l4j.ini file.

A popular way to store Unicode documents is in UTF-8, which is a superset of ASCII (but can still store all Unicode data); if you get an error message about document I/O during reading, try setting the encoding to UTF-8, or some other locally popular encoding.

Chapter 4
CREOLE: the GATE Component Model [#]

The GATE architecture is based on components: reusable chunks of software with well-defined interfaces that may be deployed in a variety of contexts. The design of GATE is based on an analysis of previous work on infrastructure for LE, and of the typical types of software entities found in the fields of NLP and CL (see in particular chapters 4–6 of [Cunningham 00]). Our research suggested that a profitable way to support LE software development was an architecture that breaks down such programs into components of various types. Because LE practice varies very widely (it is, after all, predominantly a research field), the architecture must avoid restricting the sorts of components that developers can plug into the infrastructure. The GATE framework accomplishes this via an adapted version of the Java Beans component framework from Sun, as described in section 4.2.

GATE components may be implemented by a variety of programming languages and databases, but in each case they are represented to the system as a Java class. This class may do nothing other than call the underlying program, or provide an access layer to a database; on the other hand it may implement the whole component.

GATE components are one of three types:

• LanguageResources (LRs) represent entities such as lexicons, corpora or ontologies;

• ProcessingResources (PRs) represent entities that are primarily algorithmic, such as parsers, generators or ngram modellers;

• VisualResources (VRs) represent visualisation and editing components that participate in GUIs.

The distinction between language resources and processing resources is explored more fully in section D.1.1. Collectively, the set of resources integrated with GATE is known as CREOLE: a Collection of REusable Objects for Language Engineering.

In the rest of this chapter:

• Section 4.3 describes the lifecycle of GATE components;

• Section 4.4 describes how Processing Resources can be grouped into applications;

• Section 4.5 describes the relationship between Language Resources and their datastores;

• Section 4.6 summarises GATE’s set of built-in components;

• Section 4.7 describes how configuration data for Resource types is supplied to GATE.

4.1 The Web and CREOLE [#]

GATE allows resource implementations and Language Resource persistent data to be distributed over the Web, and uses Java annotations for configuration of resources (and GATE itself).

Resource implementations are grouped together as ‘plugins’, stored either in a single JAR file published via the standard Maven repository mechanism, or at a URL (when the resources are in the local file system this would be a file:/ URL). When a plugin is loaded into GATE it looks for a configuration file called creole.xml relative to the plugin URL or inside the plugin JAR file and uses the contents of this file in combination with Java annotations on the source code to determine what resources this plugin declares and, in the case of directory-style plugins, where to find the classes that implement the resource types (typically a JAR file in the plugin directory). GATE retrieves the configuration information from the plugin’s resource classes and adds the resource definitions to the CREOLE register. When a user requests an instantiation of a resource, GATE creates an instance of the resource class in the virtual machine.

Language resource data can be stored in binary serialised form in the local file system.

4.2 The GATE Framework [#]

We can think of the GATE framework as a backplane into which users can plug CREOLE components. The user gives the system a list of plugins to search when it starts up, and components in those plugins are loaded by the system.

The backplane performs these functions:

• component discovery, bootstrapping, loading and reloading;

• management and visualisation of native data structures for common information types;

• generalised data storage and process execution.

A set of components plus the framework is a deployment unit which can be embedded in another application.

At their most basic, all GATE resources are Java Beans, the Java platform’s model of software components. Beans are simply Java classes that obey certain interface conventions:

• beans must have no-argument constructors.

• beans have properties, defined by pairs of methods named by the convention setProp and getProp .

GATE uses Java Beans conventions to construct and configure resources at runtime, and defines interfaces that different component types must implement.

4.3 The Lifecycle of a CREOLE Resource [#]

CREOLE resources exhibit a variety of forms depending on the perspective they are viewed from. Their implementation is as a Java class plus an XML metadata file living at the same URL. When using GATE Developer, resources can be loaded and viewed via the resources tree (left pane) and the ‘create resource’ mechanism. When programming with GATE Embedded, they are Java objects that are obtained by making calls to GATE’s Factory class. These various incarnations are the phases of a CREOLE resource’s ‘lifecycle’. Depending on what sort of task you are using GATE for, you may use resources in any or all of these phases. For example, you may only be interested in getting a graphical view of what GATE’s ANNIE Information Extraction system (see Chapter 6) does; in this case you will use GATE Developer to load the ANNIE resources, and load a document, and create an ANNIE application and run it on the document. If, on the other hand, you want to create your own resources, or modify the Java code of an existing resource (as opposed to just modifying its grammar, for example), you will need to deal with all the lifecycle phases.

The various phases may be summarised as:

Creating a new resource from scratch (bootstrapping).

To create the binary image of a resource (a Java class in a JAR file), and the XML file that describes the resource to GATE, you need to create the appropriate .java file(s), compile them and package them as a .jar. GATE provides a Maven archetype to start this process – see Section 7.12. Alternatively you can simply copy code from an existing resource.

Instantiating a resource in GATE Embedded.

To create a resource in your own Java code, use GATE’s Factory class (this takes care of parameterising the resource, restoring it from a database where appropriate, etc. etc.). Section 7.2 describes how to do this.

Loading a resource into GATE Developer.

To load a resource into GATE Developer, use the various ‘New ... resource’ options from the File menu and elsewhere. See Section 3.1.

Resource configuration and implementation.

GATE’s Maven archetype will create an empty resource that does nothing. In order to achieve the behaviour you require, you’ll need to change the Java code and its configuration annotations. See section 4.7 for more details.

4.4 Processing Resources and Applications [#]

PRs can be combined into applications. Applications model a control strategy for the execution of PRs. In GATE, applications are called ‘controllers’ accordingly.

Currently the main application types provided by GATE implement sequential or “pipeline” control flow. There are two main types of pipeline:

Simple pipelines

simply group a set of PRs together in order and execute them in turn. The implementing class is called SerialController.

Corpus pipelines

are specific for LanguageAnalysers – PRs that are applied to documents and corpora. A corpus pipeline opens each document in the corpus in turn, sets that document as a runtime parameter on each PR, runs all the PRs on the corpus, then closes the document. The implementing class is called SerialAnalyserController.

Conditional versions of these controllers are also available. These allow processing resources to be run conditionally on document features. See Section 3.8.2 for how to use these. If more flexibility is required, the Groovy plugin provides a scriptable controller (see section 7.16.3) whose execution strategy is specified using the Groovy programming language.

Controllers are themselves PRs – in particular a simple pipeline is a standard PR and a corpus pipeline is a LanguageAnalyser – so one pipeline can be nested in another. This is particularly useful with conditional controllers to group together a set of PRs that can all be turned on or off as a group.

There is also a real-time version of the corpus pipeline. When creating such a controller, a timeout parameter needs to be set which determines the maximum amount of time (in milliseconds) allowed for the processing of a document. Documents that take longer to process, are simply ignored and the execution moves to the next document after the timeout interval has lapsed.

All controllers have special handling for processing resources that implement the interface gate.creole.ControllerAwarePR. This interface provides methods that are called by the controller at the start and end of the whole application’s execution – for a corpus pipeline, this means before any document has been processed and after all documents in the corpus have been processed, which is useful for PRs that need to share data structures across the whole corpus, build aggregate statistics, etc. For full details, see the JavaDoc documentation for ControllerAwarePR.

4.5 Language Resources and Datastores [#]

Language Resources can be stored in Datastores. Datastores are an abstract model of disk-based persistence, which can be implemented by various types of storage mechanism. Here are the types implemented:

Serial Datastores

are based on Java’s serialisation system, and store data directly into files and directories.

Lucene Datastores

is a full-featured annotation indexing and retrieval system. It is provided as part of an extension of the Serial Datastores. See Section 9 for more details.

4.6 Built-in CREOLE Resources [#]

GATE comes with various built-in components:

• Language Resources modelling Documents and Corpora, and various types of Annotation Schema – see Chapter 5.

• Processing Resources that are part of the ANNIE system – see Chapter 6.

• Gazetteers – see Chapter 13.

• Ontologies – see Chapter 14.

• Machine Learning resources – see Chapter 19.

• Alignment tools – see Chapter 20.

• Parsers and taggers – see Chapter 18.

• Other miscellaneous resources – see Chapter 23.

4.7 CREOLE Resource Configuration [#]

This section describes how to supply GATE with the configuration data it needs about a resource, such as what its parameters are, how to display it if it has a visualisation, etc. Several GATE resources can be grouped into a single plugin, which is a directory or JAR file containing an XML configuration file called creole.xml at its root. The creole.xml file provides metadata about the plugin as a whole, the configuration for individual resource classes is given directly in the Java source file using Java annotations.

A creole.xml file has a root element <CREOLE-DIRECTORY> which supports several optional attribute:

NAME:

The name of the plugin. Used in the GUI to help identify the plugin in a nicer way than the direcory or artifact name.

VERSION:

The version number of the plugin. For example, 3, 3.1, 3.11, 3.12-SNAPSHOT etc.

DESCRIPTION:

A short description of the resources provided by the plugin. Note that there is really only space for a single sentence in the GUI.

GATE-MIN:

The earliest version of GATE that this plugin is compatible with. This should be in the same format as the version shown in the GATE titlebar, i.e. 8.5 or 8.6-beta1. Do not include the build number information.

Currently all these attributes are optional, as in most cases the information can be pulled from other elements of the plugin metadata; for example, plugins distributued via Maven will use information from the pom.xml if not specified.

For many simple single-JAR plugins the creole.xml file need have no other content – just an empty <CREOLE-DIRECTORY /> element – but there are certain child elements that are used in some types of plugin.

Directory-style plugins need at least one <JAR> child element to tell GATE where to find the classes that implement the plugin’s resources. Each <JAR> element contains a path to a JAR file, which is resolved relative to the location of the creole.xml, for example:

<CREOLE-DIRECTORY>
  <JAR SCAN="true">myPlugin.jar</JAR>
  <JAR>lib/thirdPartyLib.jar</JAR>
</CREOLE-DIRECTORY>

JAR files that contain resource classes must be specified with SCAN="true", which tells GATE to scan the JAR contents to discover resource classes annotated with @CreoleResource (see below). Other JAR files required by the plugin can be specified using other <JAR> elements without SCAN="true".

Plugins can depend on other plugins, for example if a plugin defines a PR which internally makes use of a JAPE transducer then that plugin would declare that it depends on ANNIE (the standard plugin that defines the JAPE transducer PR). This is done with a <REQUIRES> element. To depend on a single-JAR plugin from a Maven repository, use an empty element with attributes GROUP, ARTIFACT and VERSION, for example

<CREOLE-DIRECTORY>
  <REQUIRES GROUP="uk.ac.gate.plugins"
            ARTIFACT="annie"
            VERSION="8.5" />
</CREOLE-DIRECTORY>

Directory-style plugins can also depend on other directory-style plugins using a relative path (e.g. <REQUIRES>../other-plugin</REQUIRES>, but this is generally discouraged – if your plugin is likely to be required as a dependency of other plugins then it is better converted to the single JAR Maven style so the dependency can be handled via group/artifact/version co-ordinates.

You may see old plugins with other elements such as <RESOURCE>, this is the older style of configuration in XML, which is now deprecated in favour of the annotations described below.

4.7.1 Configuring Resources using Annotations [#]

The configuration of the resources within a plugin is handled using Java annotation types to embed the configuration data directly in the Java source code. @CreoleResource is used to mark a class as a GATE resource, and parameter information is provided through annotations on the JavaBean set methods. At runtime these annotations are read and used to construct the resource data that is registered with the CREOLE register. The metadata annotation types are all marked @Documented so the CREOLE configuration data will be visible in the generated JavaDoc documentation.

For more detailed information, see the JavaDoc documentation for gate.creole.metadata.

Basic Resource-Level Data

To mark a class as a CREOLE resource, simply use the @CreoleResource annotation (in the gate.creole.metadata package), for example:

The @CreoleResource annotation provides slots for various configuration values:

name

(String) the name of the resource, as it will appear in the ‘New’ menu in GATE Developer. If omitted, defaults to the bare name of the resource class (without a package name).

comment

(String) a descriptive comment about the resource, which will appear as the tooltip when hovering over an instance of this resource in the resources tree in GATE Developer. If omitted, no comment is used.

helpURL

(String) a URL to a help document on the web for this resource. It is used in the help browser inside GATE Developer.

isPrivate

(boolean) should this resource type be hidden from the GATE Developer GUI, so it does not appear in the ‘New’ menus? If omitted, defaults to false (i.e. not hidden).

icon

(String) the icon to use to represent the resource in GATE Developer. If omitted, a generic language resource or processing resource icon is used. The value of this element can be:

• a plain name such as “Application”, which is prepended with the package name gate.resources.img.svg. and the suffix “Icon”, which is assumed to be a Java class implementing javax.swing.Icon. GATE provides a collection of these icon classes which are generated from SVG files and are fully scalable for high-DPI monitors.

• a path to an image file inside the plugin’s JAR, starting with a forward slash, e.g. /myplugin/images/icon.png

interfaceName

(String) the interface type implemented by this resource, for example a new type of document would specify "gate.Document" here.

tool

(boolean) is this resource type a tool? The “tool” flag identifies things like resource helpers and resources that contribute items to the tools menu in GATe Developer.

autoInstances

(array of @AutoInstance annotations) definitions for any instances of this resource that should be created automatically when the plugin is loaded. If omitted, no auto-instances are created by default. Auto-instances are useful for things like document formats and tools which contribute behaviour to other GATE resources, and which should be available by default whenever the plugin is loaded.

For visual resources only, the following elements are also available:

guiType

(GuiType enum) the type of GUI this resource defines. The options are LARGE (the VR should appear in the main right-hand panel of the GUI) or SMALL (the VR should appear in the bottom left hand corner below the resources tree).

resourceDisplayed

(String) the class name of the resource type that this VR displays, e.g. "gate.Corpus". Any resource whose type is assignable to this type will be displayed with this viewer, so for example a VR that can display all types of document would specify gate.Document, whereas a VR that can only display the default GATE document implementation would specify gate.corpora.DocumentImpl.

mainViewer

(boolean) is this VR the ‘most important’ viewer for its displayed resource type? If there are several different viewers that are all applicable to a particular resource type, the mainViewer hint helps GATE Developer decide which one should be initially visible as the selected tab.

For annotation viewers, you should specify an annotationTypeDisplayed element giving the annotation type that the viewer can display (e.g. Sentence).

Resource Parameters

Parameters are declared by placing annotations on their JavaBean set methods. To mark a setter method as a parameter, use the @CreoleParameter annotation, for example:

  @CreoleParameter(comment = "The location of the list of abbreviations")
  public void setAbbrListUrl(URL listUrl) {
    ...

GATE will infer the parameter’s name from the name of the JavaBean property in the usual way (i.e. strip off the leading set and convert the following character to lower case, so in this example the name is abbrListUrl). The parameter name is not taken from the name of the method parameter. The parameter’s type is inferred from the type of the method parameter (java.net.URL in this case).

The annotation elements of @CreoleParameter are as follows:

comment

(String) an optional descriptive comment about the parameter.

defaultValue

(String) the optional default value for this parameter. The value is specified as a string but is converted to the relevant type by GATE according to the conversions described below.

suffixes

(String) for parameters of type URL or ResourceReference, a semicolon-separated list of default file suffixes that this parameter accepts.

collectionElementType

(Class) for Collection-valued parameters, the type of the elements in the collection. This can usually be inferred from the generic type information, for example public void setIndices(List<Integer> indices), but must be specified if the set method’s parameter has a raw (non-parameterized) type.

Parameter default values must be specified as strings, but parameters can be of any type and GATE applies the following rules to convert the default string into an appropriate value for the parameter type:

String

if the parameter is of type String the default value is used directly

Primitive wrapper types e.g. Integer

the string is passed to the relevant valueOf method

enum types

the value is passed to Enum.valueOf

java.net.URL or gate.creole.ResourceReference

the string is parsed as a URI, and if the URI is relative then it is resolved against the plugin (for directory-style plugins this means against the location of creole.xml and for Maven plugins it is the root of the plugin JAR file)

collection types (Set, List, etc.)

the string is treated as a semicolon-separated list of values, and each value is converted to the collection’s element type following these same rules.

gate.FeatureMap

the string is parsed as “feature1=value1;feature2=value2” etc. (a semicolon-separated list of “name=value” pairs)

any other java.* type

if the type has a constructor taking a String then that constructor is called with the default string as its parameter.

If there is no default specified, the default value is null.

Mutually-exclusive parameters are handled by adding a disjunction="label" and priority=n to the @CreoleParameter annotation – all parameters that share the same label are grouped in the same disjunction, and will be offered in order of priority. The parameter with the smallest priority value will be the one listed first, and thus the one that is offered initially when creating a resource of this type in GATE Developer. For example, the following is a simplified extract from gate.corpora.DocumentImpl:

This declares the parameters “stringContent” and “sourceUrl” as mutually-exclusive, and when creating an instance of this resource in GATE Developer the parameter that will be shown initially is sourceUrl. To set stringContent instead the user must select it from the drop-down list. Parameters with the same declared priority value will appear next to each other in the list, but their relative ordering is not specified. Parameters with no explicit priority are always listed after those that do specify a priority.

Optional and runtime parameters are marked using extra annotations, for example:

Runtime parameters apply only to Processing Resources, and are parameters that are not used when the resource is initialised but instead only when it is executed. An “optional” parameter is one that does not have to be set before creating or executing the resource.

Inheritance

A resource will inherit any configuration data that was not explicitly specified from annotations on its parent class and on any interfaces it implements. Specifically, if you do not specify a comment, interfaceName, icon, annotationTypeDisplayed or the GUI-related elements (guiType and resourceDisplayed) on your @CreoleResource annotation then GATE will look up the class tree for other @CreoleResource annotations, first on the superclass, its superclass, etc., then at any implemented interfaces, and use the first value it finds. This is useful if you are defining a family of related resources that inherit from a common base class.

The resource name and the isPrivate and mainViewer flags are not inherited.

Parameter definitions are inherited in a similar way. For example, the gate.LanguageAnalyser interface provides two parameter definitions via annotated set methods, for the corpus and document parameters. Any @CreoleResource annotated class that implements LanguageAnalyser, directly or indirectly, will get these parameters automatically.

Of course, there are some cases where this behaviour is not desirable, for example if a subclass calculates a value for a superclass parameter rather than having the user set it directly. In this case you can hide the parameter by overriding the set method in the subclass and using a marker annotation:

The overriding method will typically just call the superclass one, as its only purpose is to provide a place to put the @HiddenCreoleParameter annotation.

Alternatively, you may want to override some of the configuration for a parameter but inherit the rest from the superclass. Again, this is handled by trivially overriding the set method and re-annotating it:

Note that for backwards compatibility, data is only inherited from superclass annotations if the subclass is itself annotated with @CreoleResource.

4.7.2 Loading Third-Party Libraries in a Maven plugin [#]

A Maven plugin is distributed as a single JAR file, but if the plugin depends on any third-party libraries these can be specified as dependencies in the corresponding POM file in the usual Maven way as compile or runtime scoped dependencies.

If one plugin has a compile-time dependency on another (as opposed to simply a runtime dependency when one plugin creates resources defined in another) then you should specify the dependency in your POM as <scope>provided</scope> as well as declaring it in creole.xml with group/artifact/version.

4.8 Tools: How to Add Utilities to GATE Developer [#]

Visual Resources allow a developer to provide a GUI to interact with a particular resource type (PR or LR), but sometimes it is useful to provide general utilities for use in the GATE Developer GUI that are not tied to any specific resource type. Examples include the annotation diff tool and the Groovy console (provided by the Groovy plugin), both of which are self-contained tools that display in their own top-level window. To support this, the CREOLE model has the concept of a tool.

A resource type is marked as a tool by setting tool = true in the @CreoleResource annotation. If a resource is declared to be a tool, and written to implement the gate.gui.ActionsPublisher interface, then whenever an instance of the resource is created its published actions will be added to the “Tools” menu in GATE Developer.

Since the published actions of every instance of the resource will be added to the tools menu, it is best not to use this mechanism on resource types that can be instantiated by the user. The “tool” marker is best used in combination with the “private” flag (to hide the resource from the list of available types in the GUI) and one or more hidden autoinstance definitions to create a limited number of instances of the resource when its defining plugin is loaded. See the GroovySupport resource in the Groovy plugin for an example of this.

4.8.1 Putting Your Tools in a Sub-Menu [#]

If your plugin provides a number of tools (or a number of actions from the same tool) you may wish to organise your actions into one or more sub-menus, rather than placing them all on the single top-level tools menu. To do this, you need to put a special value into the actions returned by the tool’s getActions() method:

The key must be GateConstants.MENU_PATH_KEY and the value must be an array of strings. Each string in the array represents the name of one level of sub-menus. Thus in the example above the action would be placed under “Tools → Acme toolkit → Statistics”. If no MENU_PATH_KEY value is provided the action will be placed directly on the Tools menu.

4.8.2 Adding Tools To Existing Resource Types [#]

While Visual Resources (VR) allow you to add new features to a particular resource they have a number of shortcomings. Firstly not every new feature will require a full VR; often a new entry on the resources right-click menu will suffice. More importantly new feautres added via a VR are only available while the VR is open. A Resource Helper is a form of Tool, as above, which can add new menu options to any existing resource type without requiring a VR.

A Resource Helper is defined in the same way as a Tool (by setting the tool = true feature of the @CreoleResource annotation and loaded via an autoinstance definition) but must also extend the gate.gui.ResourceHelper class. A Resource Helper can then return a set of actions for a given resource which will be added to its right-click menu. See the FastInfosetExporter resource in the “Format: FastInfoset” plugin for an example of how this works.

A Resource Helper may also make new API calls accessable to allow similar functionality to be made available to GATE Embedded, see Section 7.19 for more details on how this works.

Chapter 5
Language Resources: Corpora, Documents and Annotations [#]

This chapter documents GATE’s model of corpora, documents and annotations on documents. Section 5.1 describes the simple attribute/value data model that corpora, documents and annotations all share. Section 5.2, Section 5.3 and Section 5.4 describe corpora, documents and annotations on documents respectively. Section 5.5 describes GATE’s support for diverse document formats, and Section 5.5.2 describes facilities for XML input/output.

5.1 Features: Simple Attribute/Value Data [#]

GATE has a single model for information that describes documents, collections of documents (corpora), and annotations on documents, based on attribute/value pairs. Attribute names are strings; values can be any Java object. The API for accessing this feature data is Java’s Map interface (part of the Collections API).

5.2 Corpora: Sets of Documents plus Features [#]

A Corpus in GATE is a Java Set whose members are Documents. Both Corpora and Documents are types of LanguageResource (LR); all LRs have a FeatureMap (a Java Map) associated with them that stored attribute/value information about the resource. FeatureMaps are also used to associate arbitrary information with ranges of documents (e.g. pieces of text) via the annotation model (see below).

Documents have a DocumentContent which is a text at present (future versions may add support for audiovisual content) and one or more AnnotationSets which are Java Sets.

5.3 Documents: Content plus Annotations plus Features [#]

Documents are modelled as content plus annotations (see Section 5.4) plus features (see Section 5.1). The content of a document can be any subclass of DocumentContent.

5.4 Annotations: Directed Acyclic Graphs [#]

Annotations are organised in graphs, which are modelled as Java sets of Annotation. Annotations may be considered as the arcs in the graph; they have a start Node and an end Node, an ID, a type and a FeatureMap. Nodes have pointers into the sources document, e.g. character offsets.

5.4.1 Annotation Schemas [#]

Annotation schemas provide a means to define types of annotations in GATE. GATE uses the XML Schema language supported by W3C for these definitions. When using GATE Developer to create/edit annotations, a component is available (gate.gui.SchemaAnnotationEditor) which is driven by an annotation schema file. This component will constrain the data entry process to ensure that only annotations that correspond to a particular schema are created. (Another component allows unrestricted annotations to be created.)

Schemas are resources just like other GATE components. Below we give some examples of such schemas. Section 3.4.6 describes how to create new schemas. Note that each schema file defines a single annotation type, however it is possible to use include definitions in a schema to refer to other schemas in order to load a whole set of schemas as a group. The default schemas for ANNIE annotation types (defined in resources/schema in the ANNIE plugin) give an example of this technique.

Date Schema

<?xml version="1.0"?>
<schema
xmlns="http://www.w3.org/2000/10/XMLSchema">
 <!-- XSchema deffinition for Date-->
  <element name="Date">
    <complexType>
      <attribute name="kind"  use="optional">
        <simpleType>
          <restriction base="string">
            <enumeration value="date"/>
            <enumeration value="time"/>
            <enumeration value="dateTime"/>
          </restriction>
        </simpleType>
    </attribute>
  </complexType>
 </element>
</schema>

Person Schema

<?xml version="1.0"?>
<schema
xmlns="http://www.w3.org/2000/10/XMLSchema">
    <!-- XSchema definition for Person-->
    <element name="Person" />
</schema>

Address Schema

<?xml version="1.0"?> <schema
xmlns="http://www.w3.org/2000/10/XMLSchema">
    <!-- XSchema definition for Address-->
    <element name="Address">
      <complexType>
        <attribute name="kind"  use="optional">
          <simpleType>
            <restriction base="string">
              <enumeration value="email"/>
              <enumeration value="url"/>
              <enumeration value="phone"/>
              <enumeration value="ip"/>
              <enumeration value="street"/>
              <enumeration value="postcode"/>
              <enumeration value="country"/>
              <enumeration value="complete"/>
            </restriction>
        </simpleType>
    </attribute>
  </complexType>
</element>
</schema>

5.4.2 Examples of Annotated Documents [#]

This section shows some simple examples of annotated documents.

This material is adapted from [Grishman 97], the TIPSTER Architecture Design document upon which GATE version 1 was based. Version 2 has a similar model, although annotations are now graphs, and instead of multiple spans per annotation each annotation now has a single start/end node pair. The current model is largely compatible with [Bird & Liberman 99], and roughly isomorphic with "stand-off markup" as latterly adopted by the SGML/XML community.

Each example is shown in the form of a table. At the top of the table is the document being annotated; immediately below the line with the document is a ruler showing the position (byte offset) of each character (see TIPSTER Architecture Design Document).

Underneath this appear the annotations, one annotation per line. For each annotation is shown its Id, Type, Span (start/end offsets derived from the start/end nodes), and Features. Integers are used as the annotation Ids. The features are shown in the form name = value.

The first example shows a single sentence and the result of three annotation procedures: tokenization with part-of-speech assignment, name recognition, and sentence boundary recognition. Each token has a single feature, its part of speech (pos), using the tag set from the University of Pennsylvania Tree Bank; each name also has a single feature, indicating the type of name: person, company, etc.

Annotations will typically be organized to describe a hierarchical decomposition of a text. A simple illustration would be the decomposition of a sentence into tokens. A more complex case would be a full syntactic analysis, in which a sentence is decomposed into a noun phrase and a verb phrase, a verb phrase into a verb and its complement, etc. down to the level of individual tokens. Such decompositions can be represented by annotations on nested sets of spans. Both of these are illustrated in the second example, which is an elaboration of our first example to include parse information. Each non-terminal node in the parse tree is represented by an annotation of type parse.