Cancer Research Use Case

Angus Roberts, Yaoyong Li, Hamish Cunningham

University of Sheffield

Participants

• University of Sheffield
• systems development
• provide end-user services built on LarKC

• WHO International Agency for Research on Cancer
• requirements capture
• design discussion
• user evaluation

• Siemens
• oversight
• advisory

Objectives

• Objectives
• use the collider platform to assist:
• researchers developing new techniques in genetic epidemiology of cancer
• carcinogenesis researchers producing reference works (monographs) on risk factors
• how can we organise and link scientific data to support cancer research?

• Solutions and technologies
• semantic annotation of scientific literature, using information extraction
• using this annotation to support complex retrieval

Scenario (1): Monograph production

The scenario

• IARC monographs are expert reviews of current knowlege of potential risk factors
• Monograph production involves
• study area setup and invitation of experts
• expert working group reviews the current state of the art
• collation, editing, and publishing of expert reviews
• Each of these steps either makes use of, or contributes to, the research literature

Scenario (1): Monograph production

Biomedical research literature: the problem

• Is large and growing exponentially
• 17 million abstracts, currently 600 000 added a year
• Involves complex domain and social and inter-relationships
• between authors and research groups
• between genes, proteins, carcinogens, diseases etc
• between the literature and a myriad of research databases
• Monograph production could benefit from tools to assist in
• mining the literature
• organising the literature
• querying the literature

Scenario (1): Monograph production

Solution: document clustering

• What axes we might connect and cluster along?
• authorship, citations, salient terms
• domain specific terminology contained and relations between terms

• Citation analysis and social network analysis
• how do authors interelate?
• which papers are most heavily cited?
• how do papers interelate thematically?

• Use of IE to provide network metrics

• Indexing and interlinking with domain-specific ontologies

Scenario (1): Monograph production

Solution: a shared, intelligent, document repository

• Monograph production is a highly social process
• Shared web access for upload of documents and notes
• On-the-fly semantic annotation of documents
• Hyperlinking betwen documents and ontologies, to support visualisation and retrieval

Scenario (2): SNP studies

The problem

• SNPs are highly conserved regions of the genome, that can be readily mapped
• Is an SNP more common in a given disease than expected?
• Use array techniques to look at over 500 000 SNPs in populations of cancer patients
• Small population sizes mean that the significance of a particular SNP might be falsely lowered
• Important SNPs may be ranked too low to be considered
• Typically, one or two important SNPs might appear in the top 20 000 ranked SNPs

Scenario (2): SNP studies

A possible solution?

• Can we introduce additional information to improve ranking techniques?
• Retrieve all papers relevant to the gene carrying an SNP
• Semantic annotation of the papers:
• genes and proteins
• anatomy
• disease
• risk factors
• Linkage to existing domain ontologies
• Query the resultant knowledge base to boost rankings with a prior probability

Semantic annotation (1)

• All of the above requires semantic annotation of the research literature
• A high level of domain terminology
• Ambiguous terminology
• Complex relations between things
• cause and effect
• molecular interactions

Semantic annotation (2)

• Semantically annotate against existing ontologies
• Biomedicine has many ontologies and knowledge resources
• widely used
• largest and most sophisticated ontologies currently in real use
• For example
• GO: Gene Ontology
• UniProt: Universal Protein Database
• UMLS: Unified Medical Language System

Information Extraction

• Semantic annotation using Information Extraction (IE)
• Text is unstructured
• IE analyses unstructured text to extract structured information
• For example, annotation of
• named entities, such as authors
• domain-specific entities, such as diseases, genes, proteins
• the relationships between these entities
• We will use the GATE toolkit for IE

GATE is...

• like Eclipse for Natural Language Engineering, like Lucene for Information Extraction, a leading toolkit for Text Mining
• used worldwide by thousands of scientists, companies, teachers and students
• used for all sorts of language processing tasks, including Information Extraction in many languages
• funded by the EPSRC, BBSRC, AHRC, the EU and commercial users
• comprised of an architecture, free open source framework (or SDK) and graphical development environment, lots of components
• standards based
• 10 years old in 2005, used in many research projects and under integration with IBM's UIMA
• based on MVC, mobile code, continuous integration, and test-driven development, with code hosted on SourceForge

GATE architecture and APIs

GATE IE: outline

• Machine learning sandwich made with symbolic bread

• Pre-processing bread: provide features for ML
• find terms using gazeteers and term recognisers
• find lexical syntactic patterns using the JAPE pattern language

• ML filling: applying classifiers to learn semantic annotations, based on the features introduced by pre-processing

• Post-processing bread: combine and manipulate learned annotations with JAPE
• combine things we have found in the text
• relate extracted information to ontologies

• Output as triples to the LarKC semantic repository

GATE IE: Pre-processing

• Standard NLP tools
• Part-of-speech tagging
• Morphological analysis
• Parsing

• Term recognition with
• FSMs over large terminologies
• ontologies

• JAPE: a pattern / action language
• LHS language to match patterns in annotation graphs
• RHS language to act on those annotations

GATE IE: Machine Learning

• GATE has a Learning API to abstract away from underlying classifier algorithms
• Used to learn
• entities
• relations
• text classification
• The API maps annotations to learning instances and to classifiers...
• ... and maps back to learned annotations
• We generally use a variant of Support Vector Machines: SVM with uneven margins
• Good performance over the sparse data typical of NLP applications

Data for training and evaluation

• Training and evaluation of IE requires pre-annotated data: a gold standard
• There are already several pre-annotated corpora of the biomedical literature
• User requirements analysis will show if these are sufficeint for our needs
• If not, we will need our own manual annotation to create a gold standard
• This is potentially labour intnesive and costly
• We will use machine-learning assisted, mixed initiative annotation
• SAFE: Semantic Annotation Factory Environment

SAFE Architecture (1)

• Documents/corpora and ontologies stored in central repository, accessed via web services
• Machine annotation handled by annotation web services which talk to the document and ontology services
• A single annotation service can be a front-end to many worker nodes, possibly distributed on multiple machines - this is transparent to the caller
• Language engineers create and deploy annotation services using GATE
• Information curator manages the document and ontology services (SAFEKeeper)
• Human annotators use a Java WebStart GUI based on the GATE document editor

SAFE Architecture (2)

• Workflow may be simple
• Call a pre-processing annotation service on each document
• Give the document to a human
• Information curator checks a sample of documents for QC
• or more complex
• For 10% of the corpus, pass the same document to two humans
• Information curator has tools in SAFEKeeper to quickly check level of agreement between the annotators.
• Annotated documents used to train an ML model
• When model is good enough, start making suggestions to the annotators

The scale of the problem (1)

Retraining models

• Imagine a monograph production meeting in which someone raises a whole new subject area: you need to retrain IE software to that area, apply it, and dynamically interrelate output to the existing knowledge bases.

• Scale of the literature: the Medline biomedical abstract database contains 17 million abstracts

• Can we do dynamic IE, to reflect changes in:
• requirements
• ontologies
• the literature itself

The scale of the problem (2)

Semantic annotation

• An example: time for annotation of clinical documents with entities and relationships
• 500 000 documents
• average size 1 KByte
• 40 hours on dual quad core xeon 16GB memory

• Cutting edge NLP is data and processor hungry. Building good statistical models of natural langauge requires huge numbers of examples. The more examples, the better the models.

The scale of the problem (3)

Retrieval

• The information extracted will be large and highly inter-related.

• An example from 500 000 clinical documents:
• entites per document 21.5
• relations per document 11
• potentially > 1000 triples per document
• > 500 million triples total

• IE over all 17 million Medline documents will result in trillions of triples

The scale of the problem (4)

• Interconnect all the IE output form Medline with existing ontologies
• e.g. UniProt: 260 million statements
• e.g. UMLS: approx 1 million concepts and 5 million terms, each with multile properties and relationships

• The use cases will involve reasoning and retrieval across this complex knowledge

• For example
• Given X carcinoma and Y environmental factor what set Z of related syndromes and factors occur in what distribution in the literature?

User Evaluation

• SUS: System Usability Scale
• De facto standard
• Provides evaluation compatible with the ISO usability standard
• Pre- and post- use questionaires
• 10 point Likert scale giving a score in a range of 0-100

References

GATE website: http://gate.ac.uk/ie

IARC monograph web site: http://monographs.iarc.fr/

R.J. Hung, J. McKay et al. A susceptibility locus for lung cancer maps to nicotinic acetylcholine receptor subunit genes on 15q25. Nature 452, pages 633-637, 2008.

The Wellcome Trust Case Control Consortium. 2007. Genome-wide association study of 14,000 cases of seven common diseases and 3,000 shared controls. Nature 447, pages 661-677, 2007.

H. Cunningham. Information Extraction, Automatic. Encyclopedia of Language and Linguistics, 2nd Edition, pages 665–677, 2005.

N. Aswani, K. Bontcheva, and H. Cunningham. Mining information for instance unification. In 5th International Semantic Web Conference (ISWC2006), Athens, Georgia, USA, 2006.

Y. Li, K. Bontcheva, and H. Cunningham. SVM Based Learning System For Information Extraction. In M. N. J. Winkler and N. Lawerence, editors, Deterministic and Statistical Methods in Machine Learning, LNAI 3635, pages 319–339. Springer Verlag, 2005.

Y. Li, K. Bontcheva, and H. Cunningham. Hierarchical, Perceptron-like Learning for Ontology Based Information Extraction. In 16th International World Wide Web Conference (WWW2007), pages 777–786, May 2007.

A. Funk, V. Tablan, K. Bontcheva, H. Cunningham, B. Davis, and S. Handschuh. Clone: Controlled language for ontology editing. In Proceedings of the 6th International Semantic Web Conference (ISWC 2007), Busan, Korea, November 2007.

P. Pantel. Data Catalysis: Facilitating Large-Scale Natural Language Data Processing. In Proceedings of the International Symposium on Universal Communication (ISUC-07), pages 201-204, Kyoto, Japan, 2007.