The GATE Crowdsourcing Plugin

Crowdsourcing is an increasingly popular, collaborative approach for acquiring annotated corpora. Despite this, reuse of corpus conversion tools and user interfaces between projects is still problematic, since these are not generally made available.

The new, open-source GATE Crowdsourcing plugin offers infrastructural support for mapping documents to crowdsourcing units and back, as well as automatically generating reusable crowdsourcing interfaces for NLP classification and sequence annotation tasks. The latest snapshot releases also offer support for automatic adjudication of results, building a consensus annotation set from the cases where enough annotators agree and gathering disputed results together for manual review.

1. Resources provided

The GATE Crowdsourcing plugin provides tools for two main types of crowdsourcing tasks:

• annotation: present crowdworkers with a snippet of text (e.g. a sentence) and ask them to mark all the mentions of a particular annotation type.
• classification: present crowdworkers with a snippet of text containing an existing annotation with several possible labels (e.g. meanings of a word), and ask them to select the most appropriate label (or “none of the above”).

The initial release of the Crowd_Sourcing plugin was part of GATE Developer 8.0, and subsequent updates including the automatic adjudication tools are available in our nightly snapshot builds.

2. Documentation

The user guide documentation for this plugin is now also available, including also screen shots.

2.1. The Basics

The GATE crowdsourcing tools are based on the CrowdFlower platform2. To get the most out of the GATE tools it is first necessary to understand a few pieces of CrowdFlower terminology.

a job is the container which represents a single end-to-end crowdsourcing process. It defines the input form you want to present to your workers, and holds a number of units of work. a unit is a single item of work, i.e. a single snippet (for annotation jobs) or a single entity (for classification jobs). CrowdFlower presents several units at a time to the user as a single task, and users are paid for each task they successfully complete. a gold unit is one where the correct answer is already known in advance. Gold units are the basis for determining whether a task has been completed “successfully” – when a job includes gold units, CrowdFlower includes one gold unit in each task but does not tell the user which one it is, and if they get the gold unit wrong then the whole task is disregarded. You can track users’ performance through the CrowdFlower platform and ignore results from users who get too many gold units wrong. CrowdFlower provides a web interface to build jobs in a browser, and also a REST API for programmatic access. The GATE tools use the REST API, so you will need to sign up for a CrowdFlower account and generate an API key which you will use to configure the various processing resources. To access the GATE crowdsourcing tools, you must first load the Crowd_Sourcing plugin. This plugin provides four PR types, a “job builder”, “results importer” and “consensus builder” for each of the two supported styles of crowdsourcing job.

2.2. Entity classification

The “entity classification” job builder and results importer PRs are intended for situations where you have pre-annotated entities but each entity could have one of several different labels. Examples could be:

a term recognintion system that has established which spans of text are candidate terms but not what class of term each annotation represents. annotation with respect to an ontology, when the same string could match one of several different ontology concepts. In the first case, the set of available labels would be constant, with the same set of options presented for every unit. In the second case each annotation would supply its own set of options (there may also be “common options” available for every annotation, such as “none of the above”).

Once you have imported the human judgments from CrowdFlower back into GATE annotations, a common next step is to take the multiply-annotated entities and attempt to build a single “consensus” set of the classifications where enough of the annotators agree. The simplest form of automatic adjudication is the majority-vote method, where classifications are automatically accepted if at least n out of m annotators agree on them. Entities that do not have the required level of agreement cannot be accepted automatically and must be double checked somehow, either directly within GATE Developer or via a second round of crowdsourcing.

When run over a corpus, the PR will look at each of the original entity annotations in turn and count up the number of annotators who selected each of the available options (including common options specified at the job level). If there is exactly one option on which at least minimumAgreement annotators agree, then it will create a single annotation in the consensus annotation set, whose type is resultAnnotationType and whose answerFeatureName feature is the selected option. If there is no agreement (either no option meets the threshold, or more than one option in the case when the threshold is below 50%), then the PRs action is determined in one of two ways.

• Resolve locally: all the judgment annotations are copied from the result set into the dispute set, so they can be inspected locally in GATE Developer (typically using the annotation stack view).
• Re-annotate by crowd: the intuition here is to gather more judgments. the original entity annotation is copied to the dispute set, with its options feature restricted to the subset of the original options that were selected by at least one of the annotators in the first round of annotation.

In the latter case, the resulting entity annotations are suitable to be imported into CrowdFlower again for another round of crowdsourced annotation, but this time highly ambiguous entities have a smaller set of choices that will be presented to the workers.

2.3. Entity annotation

The “entity annotation” job builder and results importer PRs are intended for situations where you want people to mark occurrences of named entities in plain text. A number of simplifying assumptions are made to make this task suitable for crowdsourcing:

Text is presented in short snippets (e.g. one sentence or Tweet at a time). Each job focuses on one specific entity type (if you want to annotate different entities you can do this by running a number of different jobs over the same corpus). Entity annotations are constrained to whole tokens only, and there are no adjacent annotations (i.e. a contiguous sequence of marked tokens represents one target annotation, and different annotations must be separated by at least one intervening token). This is a reasonable assumption to make given the previous point, as adjacent entities of the same type will usually be separated by something (a comma, the word “and”, etc.).

Once you have imported the human judgments from CrowdFlower back into GATE annotations, a common next step is to take the multiply-annotated entities and attempt to build a single “consensus” set of the annotations where enough of the annotators agree. The simplest form of automatic adjudication is the majority-vote method, where annotations are automatically accepted if at least n out of m annotators agree on them. Entities that do not have the required level of agreement cannot be accepted automatically and must be double checked manually.

The paper describing the plugin, which includes friendly screenshots and an idea of the crowdsourcing workflow, can be found here: The GATE Crowdsourcing Plugin: Crowdsourcing Annotated Corpora Made Easy.

3. How to cite

Please acknowledge the GATE Crowdsourcing Plugin if you have found it useful in your work. Use this reference:

K. Bontcheva, I. Roberts, L. Derczynski, D. Rout. 2014. "The GATE Crowdsourcing Plugin: Crowdsourcing Annotated Corpora Made Easy". In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL) , ACL.

Or, if you prefer BiBTeX:

@inproceedings{gate-crowd,
    title = {The GATE Crowdsourcing Plugin: Crowdsourcing Annotated Corpora Made Easy,
    author = {Bontcheva, Kalina and Roberts, Ian and Derczynski, Leon and Rout, Dominic},
    year = {2014},
    booktitle = {Proceedings of Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL)},
    publisher = {Association for Computational Linguistics},
    pages     = {97--100}
}