Chapter 2
Change Log [#]

This chapter lists major changes to GATE in roughly chronological order by release. Changes in the documentation are also referenced here.

2.1 Version 5.0-beta1 (October 2008)

2.1.1 Major new features

JAPE language improvements

Several new extensions to the JAPE language to support more flexible pattern matching. Full details are in chapter 7 but briefly:

• Negative constraints, that prevent a rule from matching if certain other annotations are present (section 7.4).
• Additional matching operators for feature values, so you can now look for {Token.length < 5}, {Lookup.minorType != "ignore"}, etc. as well as simple equality (section 7.1).
• “Meta-property” accessors, to permit access to the string covered by an annotation, the length of the annotation, etc., e.g. {Lookup@length > 4}.
• Contextual operators, allowing you to search for one annotation contained within (or containing) another, e.g. {Sentence contains {Lookup.majorType == "location"}} (see section 7.1.4).
• Additional Kleene operator for ranges, e.g. ({Token})[2,5] matches between 2 and 5 consecutive tokens.
• Additional operators can be added via runtime configuration.

Some of these extensions are similar to, but not the same as, those provided by the Montreal Transducer plugin. If you are already familiar with the Montreal Transducer, you should first look at section 7.11 which summarises the differences.

Resource configuration via Java 5 Annotations

Introduced an alternative style for supplying resource configuration information via Java 5 annotations rather than in creole.xml. The previous approach is still fully supported as well, and the two styles can be freely mixed. See section 4.9 for full details.

Ontology-based Gazetteer

Added a new plugin Ontology_Based_Gazetteer, which contains OntoRoot Gazetteer – a dynamically created gazetteer which is, in combination with few other generic GATE resources, capable of producing ontology-aware annotations over the given content with regards to the given ontology. For more details see section 9.31.

Inter-annotator agreement and merging

New plugins to support tasks involving several annotators working on the same annotation task on the same documents. The “iaaPlugin” (section 13.6) computes inter-annotator agreement scores between the annotators, and the “annotationMerging” plugin (section 9.30) merges annotations from multiple annotators into a single “consensus” annotation set.

GUI improvements

Many improvements to the GUI including context-sensitive help, a new schema-driven tool to streamline manual annotation tasks (see section 3.19.1), and many smaller fixes and user-friendliness changes.

2.1.2 Other new features and improvements

• New parser plugins: A new plugin for the Stanford Parser (see section 9.13) and a rewritten plugin for the RASP NLP tools (section 9.11).
• A new sentence splitter, based on regular expressions, has been added to the ANNIE plugin. More details in section 8.4.
• “Real-time” corpus controller (section 4.6), which terminates processing of a document if it takes longer than a configurable timeout..
• Improved alignment editor (chapter 12) with several advanced features and an API for adding your own actions to the editor.
• New features in the ML API to produce an n-gram based language model from a corpus and a so-called “document-term matrix” (see section 9.19). Also introduced features to support active learning, and various optimisations. Full details in chapter 11.
• Added new “getCovering” method to AnnotationSet. This method returns annotations that completely span the provided range. An optional annotation type parameter can be provided to further limit the returned set.

2.1.3 Specific bug fixes

• HTML document format parser: several bugs fixed, including a null pointer exception if the document contained certain characters illegal in HTML (#1754749). Also, the HTML parser now respects the “Add space on markup unpack” configuration option – previously it would always add space, even if the option was set to false.
• JAPE did not always correctly handle the case when the input and output annotation sets for a transducer were different. This has now been fixed.

Plus many more minor bug fixes

2.2 Version 4.0 (July 2007)

2.2.1 Major new features

ANNIC

ANNotations In Context: a full-featured annotation indexing and retrieval system designed to support corpus querying and JAPE rule authoring. It is provided as part of an extention of the Serial Datastores, called Searchable Serial Datastore (SSD). See section 9.29 for more details.

New machine learning API

A brand new machine learning layer specifically targetted at NLP tasks including text classification, chunk learning (e.g. for named entity recognition) and relation learning. See chapter 11 for more details.

Ontology API

A new ontology API, based on OWL In Memory (OWLIM), which offers a better API, revised ontology event model and an improved ontology editor to name but few. See chapter 10 for more details.

OCAT

Ontology-based Corpus Annotation Tool to help annotators to manually annotate documents using ontologies. For more details please see section 10.9.

Alignment Tools

A new set of components (e.g. CompoundDocument, AlignmentEditor etc.) that help in building alignment tools and in carrying out cross-document processing. See chapter 12 for more details.

New HTML Parser

A new HTML document format parser, based on Andy Clark’s NekoHTML. This parser is much better than the old one at handling modern HTML and XHTML constructs, JavaScript blocks, etc., though the old parser is still available for existing applications that depend on its behaviour.

Java 5.0 support

GATE now requires Java 5.0 or later to compile and run. This brings a number of benefits:

• Java 5.0 syntax is now available on the right hand side of JAPE rules with the default Eclipse compiler. See section B.8 for details.
• enum types are now supported for resource parameters. see section 3.13 for details on defining the parameters of a resource.
• AnnotationSet and the CreoleRegister take advantage of generic types. The AnnotationSet interface is now an extension of Set<Annotation> rather than just Set, which should make for cleaner and more type-safe code when programming to the API, and the CreoleRegister now uses parameterized types, which are backwards-compatible but provide better type-safety for new code.

2.2.2 Other new features and improvements

• Hiding the view for a particular resource (by right clicking on its tab and selecting “Hide this view”) will now completely close the associated viewers and dispose them. Re-selecting the same resource at a later time will lead to re-creating the necessary viewers and displaying them. This has two advantages: firstly it offers a mechanism for disposing views that are not needed any more without actually closing the resource and secondly it provides a way to refresh the view of a resource in the situations where it becomes corrupted.
• The DataStore viewer now allows multiple selections. This lets users load or delete an arbitrarily large number of resources in one operation.
• The Corpus editor has been completely overhauled. It now allows re-ordering of documents as well as sorting the document list by either index or document name.
• Support has been added for resource parameters of type gate.FeatureMap, and it is also possible to specify a default value for parameters whose type is Collection, List or Set. See section 3.13 for details.
• (Feature Request #1446642) After several requests, a mechanism has been added to allow overriding of GATE’s document format detection routine. A new creation-time parameter mimeType has been added to the standard document implementation, which forces a document to be interpreted as a specific MIME type and prevents the usual detection based on file name extension and other information. See section 6.5.1 for details.
• A capability has been added to specify arbitrary sets of additional features on individual gazetteer entries. These features are passed forward into the Lookup annotations generated by the gazetteer. See section 8.2 for details.
• As an alternative to the Google plugin, a new plugin called yahoo has been added to GATE to allow users to submit their query to the Yahoo search engine and to load the found pages as GATE documents. See section 9.22 for more details.
• It is now easier to run a corpus pipeline over a single document in the GATE GUI – documents now provide a right-click menu item to create a singleton corpus containing just this document. See section 3.12.1 for details.
• A new interface has been added that lets PRs receive notification at the start and end of execution of their containing controller. This is useful for PRs that need to do cleanup or other processing after a whole corpus has been processed. See section 4.6 for details.
• The GATE GUI does not call System.exit() any more when it is closed. Instead an effort is made to stop all active GATE threads and to release all GUI resources, which leads to the JVM exiting gracefully. This is particularly useful when GATE is embedded in other systems as closing the main GATE window will not kill the JVM process any more.
• The set of AnnotationSchemas that used to be included in the core gate.jar and laoded as builtins have now been moved to the ANNIE plugin. When the plugin is loaded, the default annotation schemas are instantiated automatically and are available when doing manual annotation.
• There is now support in creole.xml files for automatically creating instances of a resource that are hidden (i.e. do not show in the GUI). One example of this can be seen in the creole.xml file of the ANNIE plugin where the default annotation schemas are defined.
• A couple of helper classes have been added to assist in using GATE within a Spring application. Section 3.27 explains the details.
• Improvements have been made to the thread-safety of some internal GATE components, which mean that it is now safe to create resources in multiple threads (though it is not safe to use the same resource instance in more than one thread). This is a big advantage when using GATE in a multithreaded environment, such as a web application. See section 3.29 for details.
• Plugins can now provide custom icons for their PRs and LRs in the plugin JAR file. See section 3.13 for details.
• It is now possible to override the default location for the saved session file using a system property. See section 3.3 for details.
• The TreeTagger plugin supports a system property to specify the location of the shell interpreter used for the tagger shell script. In combination with Cygwin this makes it much easier to use the tagger on Windows. See section 9.7 for details.
• The Buchart plugin has been removed. It is superseded by SUPPLE, and instructions on how to upgrade your applications from Buchart to SUPPLE are given in section 9.12. The probability finder plugin has also been removed, as it is no longer maintained.
• The bootstrap wizard now creates a basic plugin that builds with Ant. Since a Unix-style make command is no longer required this means that the generated plugin will build on Windows without needing Cygwin or MinGW.
• The GATE source code has moved from CVS into Subversion. See section 3.2.3 for details of how to check out the code from the new repository.
• An optional parameter, keepOriginalMarkupsAS, has been added to the DocumentReset PR which allows users to decide whether to keep the Original Markups AS or not while reseting the document. See section 9.1 for more details.

2.2.3 Bug fixes and optimizations

• The Morphological Analyser has been optimized. A new FSM based, although with minor alteration to the basic FSM algorithm, has been implemented to optimize the GATE Morphological Analyser. The previous profiling figures show that the morpher when integrated with ANNIE application used to take upto 60% of the overall processing time. The optimized version only takes 7.6% of the total processing time. See section 9.9 for more details on the morpher.
• The ANNIE Sentence Splitter was optimised. The new version is about twice as fast as the previous one. The actual speed increase varies widely depending on the nature of the document.
• The imlementation of the OrthoMatcher component has been improved. This resources takes significantly less time on large documents.
• The implementation of AnnotationSets has been improved. GATE now requires up to 40% less memory to run and is also 20% faster on average. The get methods of AnnotationSet return instances of ImmutableAnnotationSet. Any attempt at modifying the content of these objects will trigger an Exception. An empty ImmutableAnnotationSet is returned instead of null.
• The Chemistry tagger (section 9.16) has been updated with a number of bugfixes and improvements.
• The Document user interface has been optimised to deal better with large bursts of events which tend to occur when the document that is currently displayed gets modified. The main advantages brought by this new implementation are:
  • The document UI refreshes faster than before.
  • The presence of the GUI for a document induces a smaller performance penalty than it used to. Due to a better threading implementation, machines benefiting from multiple CPUs (e.g. dual CPU, dual core or hyperthreading machines) should only see a negligible increase in processing time when a document is displayed compared to the situations where the document view is not shown. In the previous version, displaying a document while it was processed used to increase execution time by an order of magnitude.
  • The GUI is more responsive now when a large number of annotations are displayed, hidden or deleted.
  • The strange exceptions that used to occur occasionally while working with the document GUI should not happen any more.

And as always there are many smaller bugfixes too numerous to list here...

2.3 Version 3.1 (April 2006)

2.3.1 Major new features

Support for UIMA

UIMA (http://www.research.ibm.com/UIMA/) is a language processing framework developed by IBM. UIMA and GATE share some functionality but are complementary in most respects. GATE now provides an interoperability layer to allow UIMA applications to include GATE components in their processing and vice-versa. For full information, see chapter 16.

New Ontology API

The ontology layer has been rewritten in order to provide an abstraction layer between the model representation and the tools used for input and output of the various representation formats. An implementation that uses Jena 2 (http://jena.sourceforge.net/ontology) for reading and writing OWL and RDF(S) is provided.

Ontotext Japec Compiler

Japec is a compiler for JAPE grammars developed by Ontotext Lab. It has some limitations compared to the standard JAPE transducer implementation, but can run JAPE grammars up to five times as fast. By default, GATE still uses the stable JAPE implementation, but if you want to experiment with Japec, see section 9.28.

2.3.2 Other new features and improvements

• Addition of a new JAPE matching style ”all”. This is similar to Brill, but once all rules from a given start point have matched, the matching will continue from the next offset to the current one, rather than from the position in the document where the longest match finishes. More details can be found in Section 7.3.
• Limited support for loading PDF and Microsoft Word document formats. Only the text is extracted from the documents, no formatting information is preserved.
• The Buchart parser has been deprecated and replaced by a new plugin called SUPPLE - the Sheffield University Prolog Parser for Language Engineering. Full details, including information on how to move your application from Buchart to SUPPLE, is in section 9.12.
• The Hepple POS Tagger is now open-source. The source code has been included in the GATE distribution, under src/hepple/postag. More information about the POS Tagger can be found in Section 8.5.
• Minipar is now supported on Windows. minipar-windows.exe, a modified version of pdemo.cpp is added under the gate/plugins/minipar directory to allow users to run Minipar on windows platform. While using Minipar on Windows, this binary should be provided as a value for miniparBinary parameter. For full information on Minipar in GATE, see section 9.10.
• The XmlGateFormat writer(Save As Xml from GATE GUI, gate.Document.toXml() from GATE API) and reader have been modified to write and read GATE annotation IDs. For backward compatibility reasons the old reader has been kept. This change fixes a bug which manifested in the following situation: If a GATE document had annotations carrying features of which values were numbers representing other GATE annotation IDs, after a save and a reload of the document to and from XML, the former values of the features could have become invalid by pointing to other annotations. By saving and restoring the GATE annotation ID, the former consistency of the GATE document is maintained. For more information, see Section 6.5.2.
• The NP chunker and chemistry tagger plugins have been updated. Mark Greenwood has relicenced them under the LGPL, so their source code has been moved into the GATE distribution. See sections 9.3 and 9.16 for details.
• The Tree Tagger wrapper has been updated with an option to be less strict when characters that cannot be represented in the tagger’s encoding are encountered in the document. Details are in section 9.7.
• JAPE Transducers can be serialized into binary files. The option to load serialized version of JAPE Transducer (an init-time parameter binaryGrammarURL) is also implemented which can be used as an alternative to the parameter grammarURL. More information can be found in Section 7.9.
• On Mac OS, GATE now behaves more ‘naturally’. The application menu items and keyboard shortcuts for About and Preferences now do what you would expect, and exiting GATE with command-Q or the Quit menu item properly saves your options and current session.
• Updated versions of Weka(3.4.6) and Maxent(2.4.0).
• Optimisation in gate.creole.ml: the conversion of AnnotationSet into ML examples is now faster.
• It is now possible to create your own implementation of Annotation, and have GATE use this instead of the default implementation. See AnnotationFactory and AnnotationSetImpl in the gate.annotation package for details.

2.3.3 Bug fixes

• The Tree Tagger wrapper has been updated in order to run under Windows. See 9.7.
• The SUPPLE parser has been made more user-friendly. It now produces more helpful error messages if things go wrong. Note that you will need to update any saved applications that include SUPPLE to work with this version - see section 9.12 for details.
• Miscellaneous fixes in the Ontotext JapeC compiler.
• Optimization : the creation of a Document is much faster.
• Google plugin: The optional pagesToExclude parameter was causing a NullPointerException when left empty at run time. Full details about the plugin functionality can be found in section 9.21.
• Minipar, SUPPLE, TreeTagger: These plugins that call external processes have been fixed to cope better with path names that contain spaces. Note that some of the external tools themselves still have problems handling spaces in file names, but these are beyond our control to fix. If you want to use any of these plugins, be sure to read the documentation to see if they have any such restrictions.
• When using a non-default location for GATE configuration files, the configuration data is saved back to the correct location when GATE exits. Previously the default locations were always used.
• Jape Debugger: ConcurrentModificationException in JAPE debugger. The JAPE debugger was generating a ConcurrentModificationException during an attempt to run ANNIE. There is no exception when running without the debugger enabled. As result of fixing one unnesesary and incorrect callback to debugger was removed from SinglePhaseTransducer class.
• Plus many other small bugfixes...

2.4 January 2005

Release of version 3.

New plugins for processing in various languages (see 9.15). These are not full IE systems but are designed as starting points for further development (French, German, Spanish, etc.), or as sample or toy applications (Cebuano, Hindi, etc.).

Other new plugins:

• Chemistry Tagger 9.16
• Montreal Transducer 9.14
• RASP Parser 9.11
• MiniPar 9.10
• Buchart Parser 9.12
• MinorThird 9.25
• NP Chunker 9.3
• Stemmer 9.8
• TreeTagger 9.7
• Probability Finder
• Crawler 9.20
• Google PR 9.21

Support for SVM Light, a support vector machine implementation, has been added to the machine learning plugin (see section 9.24.7).

2.5 December 2004

GATE no longer depends on the Sun Java compiler to run, which means it will now work on any Java runtime environment of at least version 1.4. JAPE grammars are now compiled using the Eclipse JDT Java compiler by default.

A welcome side-effect of this change is that it is now much easier to integrate GATE-based processing into web applications in Tomcat. See section 3.28 for details.

2.6 September 2004

GATE applications are now saved in XML format using the XStream library, rather than by using native java serialization. On loading an application, GATE will automatically detect whether it is in the old or the new format, and so applications in both formats can be loaded. However, older versions of GATE will be unable to load applications saved in the XML format. (A java.io.StreamCorruptedException: invalid stream header exception will occcur.) It is possible to get new versions of GATE to use the old format by setting a flag in the source code. (See the Gate.java file for details.) This change has been made because it allows the details of an application to be viewed and edited in a text editor, which is sometimes easier than loading the application into GATE.

2.7 Version 3 Beta 1 (August 2004)

Version 3 incorporates a lot of new functionality and some reorganisation of existing components.

Note that Beta 1 is feature-complete but needs further debugging (please send us bug reports!).

Highlights include: completely rewritten document viewer/editor; extensive ontology support; a new plugin management system; separate .jar files and a Tomcat classloading fix; lots more CREOLE components (and some more to come soon).

Almost all the changes are backwards-compatible; some recent classes have been renamed (particularly the ontologies support classes) and a few events added (see below); datastores created by version 3 will probably not read properly in version 2. If you have problems use the mailing list and we’ll help you fix your code!

The gorey details:

• Anonymous CVS is now available. See section 3.2.3 for details.
• CREOLE repositories and the components they contain are now managed as plugins. You can select the plugins the system knows about (and add new ones) by going to ”Manage CREOLE Plugins” on the file menu.
• The gate.jar file no longer contains all the subsiduary libraries and CREOLE component resources. This makes it easier to replace library versions and/or not load them when not required (libraries used by CREOLE builtins will now not be loaded unless you ask for them from the plugins manager console).
• ANNIE and other bundled components now have their resource files (e.g. pattern files, gazetteer lists) in a separate directory in the distribution – gate/plugins.
• Some testing with Sun’s JDK 1.5 pre-releases has been done and no problems reported.
• The gate:// URL system used to load CREOLE and ANNIE resources in past releases is no longer needed. This means that loading in systems like Tomcat is now much easier.
• MAC OS X is now properly supported by the installed and the runtime.
• An Ontology-based Corpus Annotation Tool (OCAT) has been implemented as a GATE plugin. Documentation of its functionality is in Section 10.9.
• The NLG Lexical tools from the MIAKT project have now been released. See documentation in Section 9.26.
• The Features viewer/editor has been completely updated – see Sections 3.16 and 3.19 for details.
• The Document editor has been completely rewritten – see Section 3.6 for more information.
• The datastore viewer is now a full-size VR – see Section 3.21 for more information.

2.8 July 2004

GATE Documents now fire events when the document content is edited. This was added in order to support the new facility of editing documents from the GUI. This change will break backwards compatibility by requiring all DocumentListener implementations to implement a new method:
public void contentEdited(DocumentEvent e);

2.9 June 2004

A new algorithm has been implemented for the AnnotationDiff function. A new, more usable, GUI is included, and an ”Export to HTML” option added. More details about the AnnotationDiff tool are in Section 3.23.

A new build process, based on ANT (http://ant.apache.org/) is now available for GATE. The old build process, based on make, is now unsupported. See Section 3.8 for details of the new build process.

A Jape Debugger from Ontos AG has been integrated in GATE. You can turn integration ON with command line option ”-j”. If you run the GATE GUI with this option, the new menu item for Jape Debugger GUI will appear in the Tools menu. The default value of integration is OFF. We are currently awaiting documentation for this.

NOTE! Keep in mind there is ClassCastExceprion if you try to debug ConditionalCorpusPipeline. Jape Debugger is designed for Corpus Pipeline only. The Ontos code needs to be changed to allow debugging of ConditionalCorpusPipeline.

2.10 April 2004

GATE now has two alternative strategies for ontology-aware grammar transduction:

• using the [ontology] feature both in grammars and annotations; with the default Transducer.
• using the ontology aware transducer – passing an ontology LR to a new subsume method in the SimpleFeatureMapImpl. the latter strategy does not check for ontology features (this will make the writing of grammars easier – no need to specify ontology).

The changes are in:

• SinglePhaseTransducer (always call subsume with ontology – if null then the ordinary subsumption takes place)
• SimpleFeatureMapImpl (new subsume method using an ontology LR)

More information about the ontology-aware transducer can be found in Section 10.6.

A morphological analyser PR has been added to GATE. This finds the root and affix values of a token and adds them as features to that token.

A flexible gazetteer PR has been added to GATE. This performs lookup over a document based on the values of an arbitrary feature of an arbitrary annotation type, by using an externally provided gazetteer. See 9.5 for details.

2.11 March 2004

Support was added for the MAXENT machine learning library. (See 9.24.6 for details.)

2.12 Version 2.2 – August 2003

Note that GATE 2.2 works with JDK 1.4.0 or above. Version 1.4.2 is recommended, and is the one included with the latest installers.

GATE has been adapted to work with Postgres 7.3. The compatibility with PostgreSQL 7.2 has been preserved. See 3.36 for more details.

New library version – Lucene 1.3 (rc1)

A bug in gate.util.Javac has been fixed in order to account for situations when String literals require an encoding different from the platform default.

Temporary .java files used to compile JAPE RHS actions are now saved using UTF-8 and the ”-encoding UTF-8” option is passed to the javac compiler.

A custom tools.jar is no longer necessary

Minor changes have been made to the look and feel of GATE to improve its appearance with JDK 1.4.2

Some bug fixes (087, 088, 089, 090, 091, 092, 093, 095, 096 – see http://gate.ac.uk/gate/doc/bugs.html for more details).

2.13 Version 2.1 – February 2003

Integration of Machine Learning PR and WEKA wrapper (see Section 9.24).

Addition of DAML+OIL exporter.

Integration of WordNet in GATE (see Section 9.23).

The syntax tree viewer has been updated to fix some bugs.

2.14 June 2002

Conditional versions of the controllers are now available (see Section 3.15). These allow processing resources to be run conditionally on document features.

PostgreSQL Data Stores are now supported (see Section 4.7). These store data into a PostgreSQL RDBMS.

Addition of OntoGazetteer (see Section 5.2), an interface which makes ontologies visible within GATE, and supports basic methods for hierarchy management and traversal.

Integration of Protégé, so that people with developed Protégé ontologies can use them within GATE.

Addition of IR facilities in GATE (see Section 9.19).

Modification of the corpus benchmark tool (see Section 3.24), which now takes an application as a parameter.

See also for details of other recent bug fixes.