2. Future Tasks

• [TAOWeb] "what we do page" <DD,HC>:
  • A JSP that displays Di/Wim's taxonomy and allows users to select one or more concepts
  • When they hit "search" these concepts then become a keyword search against publications.dcs and the results are displayed (list of publications relevant to the set of concepts)
  • Use eprints-2.3.13.1 or later
• [TAOWeb] <HC investigate>
  • http://www.google.com/coop/cse/overview for site search?
  • New gate.ac.uk: strip the existing headers and replace with GATEWiki navigation and login etc.?
• [YAM] Yammering <HC>
  • %clone
  • yam2yam pretty printer (remove all token.image munging from the parser)
  • finish jspwiki2yam
  • html2yam
  • slidy (sale/talks/gslidy)
    • should be done like PDF and PDF dropped
    • yam layer
    • doc for team
  • finish latex support
  • Version 4 bugs:
    • the verbatim mode for HTML doesn't deal properly with angle brackets (which need to be escaped). this is due to html pre not being like latex verbatim. perhaps add another verbatim mode (%<< ?) or make chevrons TargetControls in the lexer?
    • a citation within a footnote will prematurely end the footnote. this is probably more general, and predicates don't nest. look at implementation of blockquote? prob different case
      • one option is to have a member predParser and a JAVACODE production that reads it in, counting brackets, then parses it and merges into the tree
    • both contents and includes get wrapped in paragraphs, which results in empty tags in the html. solution: use jtidy?!
    • when an image alt tag specifier is enclosed in quotation marks the generated tag puts the attribute quotes in the wrong place
    • spaces at the end of table rows (after the final |) gets discarded
    • if %contents is used multiple times, only the last one works
    • documents above around 500k (e.g. yam-huge.yam) cause out of memory errors (when fixed check if parse errors would break it again)
    • because blockquotes (and things like bold, italic) are just treated as words, they can cause overlap problems (see yam-wierd line 89 and after for example); this would be hard to fix - the workaround is to e.g. make sure to separate the blockquote markup from other constructs that it may enclose
    • an empty URL is illegal: %()
    • all list items, including empty ones, must have a space after the initial "-" or "o". (this is not a bug in fact.)
    • verbatim mode terminates lists
    • when including a file it should be possible to have the title as a higher level heading than the other headings (or simply to exclude the title)
  • Version 4 wishlist:
    • something like latex's ref command
    • definition lists
    • appendices
• [BUILD] Add sins to the cruise control build
• CLOnE for developers. A way to use CLOnE to author links into the javadocs and java2html that is generated from the code. This is to be used in developer guides to make it easy to document a system by referencing the both the core code and the test suite code.
• svn replication links: http://www.darcs.net/DarcsWiki/Tailor http://www.wandisco.com/
• [ResMan] Project management system
  • Look at x-planner: http://www.xplanner.org/
  • also http://live.gnome.org/Planner, http://www.simpleprojectmanagement.com/planner/home.html
  • also http://www.jtrac.info
• Look at integration with MindQuarry?

3. Done

• Created sins/versioning and adapted the SVNKit example code as a test suite. Successfully creates repositories and does all the usual SVN stuff on them.
• Done using Appfuse 1 because 2 not ready yet
  • Using Equinox or Appfuse 2 seed a webapp to host GATEWiki as part of the sins tree. This should be a maven multi-project build incorporating the webapp, yam and clone. Use Spring 2 / Spring MVC. Integrate with the cruise control build/test.
• Choose a testing strategy, write some initial tests (including against the test instance), add it to the Cruise Control suite (i.e. the top-level ant build) and document the build/test arrangements in the developer guide.
• Implement Spring/Acegi registration and login
• Done using Hypersonic and DAOs: Allow registration of sandboxes; store the list in a ~/.sins.xml file.
• TAO and MC sites: move to gateservice1
• Reconfigure GATEWiki to use Jetty, and add a sins/gatewiki/jetty tree.
• Investigated using Tomcat Virtual Directory: It is possible to set that pages are opened from different location and not only from Tomcat deploy (webapps) directory. The effort is adding a new .xml file where the context and a docBase is specified, so that Tomcat "knows" where to look under specified context. Different types of files could be opened (.html, .pdf, .jpg, .avi, .yam). Doesn't require Tomcat restart.
  • Jetty: since configuration is done in jetty.xml requires restart.
• GATE movies: add to DD's set, and add to SVN
• Use log4j in gate.versioning
• [GATEWiki] SVNKit integration (see also gatewiki.html#bean-interactions):
  • Enumerate functions needed from SVNKit (e.g. getting a list of what has changed when updating) and implement example tests in sins/versioning. Finalise the Sandbox object model.
  • SandboxManager: singleton that allows getting of Sandbox but is the only place Sandboxes are created; guarantees one Sandbox per disk path
  • Sandbox: prototype bean created only by SandboxManager. All locking operations must be synchronised and atomic.
• [YAM] Add logging and spring as in gate.versioning; iohandler use FSRs
• [YAM] deal with threading issues in Yam and make it a spring bean
• [GATEWiki]: Implemented representation of sandbox directory tree with Struts Menu Tag Library and a util class to work with file system. This repository tree is loaded during startup of app and set to be an attribute of ServletContext. Some problems found with Jetty when this repository is of bigger size (e.g. when I add current sins project to the registered sandboxes list). Tomcat handles this fine.
• [YAM] implement new top-level API based on YamFile
  • record generation runs and prevent concurrent identical translations
  • implement naive "needsRegen"
  • wiki links translated according to if they exist (if they don't then use the URL of a "create" page)
• [GATEWiki]:
  • Whenever asked to serve a PDF or HTML check for a more recent YAM and regenerate if necessary.
  • When requested, relative to a particular directory or file, do an SVN update or commit, tracking the resultant changes and regenerating where necessary.
  • Dynamicaly passing username/password from gatewiki to versioning:
    • Changed versioning method for adding sandboxes, so that now username and password are passed to enable svn authentification.
    • Changed gatewiki controller to pass the username and password when adding Sandbox.
    • SandboxModel table has now one more field which is reduntant but is needed and that is 'username' which is actually a username of a user who registered that sandbox.
    • We are currently assuming that users use their databases (with list of sandboxes only) localy - db is not shared. This means that only one user will register all sandboxes.
    • We also assume that svn username/password is the same for all registered sandboxes.
    • In order to user log in username/password for svn authentication, acegi password encryption feature is turned off.
• [GATEWiki] experimented with xwiki
  • building
    • svn co svn://svn.forge.objectweb.org/svnroot/xwiki/trunks-users
    • cd trunks-users/xwiki
    • ant standalone
    • cd release/xwikionjetty
    • . startxwiki.sh
    • http://localhost:8080/xwiki/bin/view/Main/WebHome
• [GATEWiki] Investigate Grails <HC>
  • grails app for wiki
    • grails 0.5 (head) install
      • svn co http://svn.codehaus.org/grails/trunk/grails/
      • set GRAILS_HOME, adjust PATH
      • it needs the install dir to be writeable
    • put in sins/clare svn
    • add sandbox domain class
    • made db persistent for development and production
• [YAM] Yammering <HC>
  • make it possible to ignore titles in included files
    • get tests working again
    • add a test for title ignore during include (includes2.yam)
    • add option and make it the default
  • create doc/yam-syntax.yam, from yam-minimal.yam
  • package.html etc.
  • document the build/test process (include check-errors, bin/install)
• [CLONE] Improvements of CLONE <DD>
  • Delete punctuation from the canonical value of quoted chunks.
  • Unparseable sentences are silently ignored now: terrible for the users! Work out how to apply rules in order with a final catch-all that returns a good warning (investigate to see if there is a smart solution to do this in JAPE).
  • Investigate possible bug that a newline sometimes needs to follow the last sentence (couldn't reproduce).
  • Support for setting different datatypes (string, date, double). 'Projects have date deadlines.' will create datatype property deadline with type date. 'Projects have numeric budget.' will create double data type for new property.
  • Support for adding synonyms to classes not only to instances added.
  • LabelIndex removed and Clie class refactored to handle methods that used to be in LabelIndex class. This reduces bug re. refreshing labels after introducing new ontology API.
  • Changed the unit test to use Ontology.toString() method. This method creates sorted string out of the ontology so that each time this method is called for the same ontology it generates the same string.
  • Make a text file to mention all possible combinations (sentences) that could be proccessed with clie. This text file is used in tests now.
  • Implemented web service for clone, and basic webapp that uses this ws to answer user questions.
• [CLONE] Querying with CLONE <DD>
  • Querying ontology with simple questions like: how many projects are there? or what are the types of documents?/what is the deadline of sekt?/list authors of 'some paper'. Look at query-example.txt for some samples.
• [CLONE]: separated query language application from clone. <DD>
• [CoW] Controllable Wiki <HC>
  • creation
    • using grails 0.5
    • grails create-app cow
    • adding to svn (http://grails.codehaus.org/Checking+Projects+into+SVN):
      • svn add cow; svn ci -m added cow; cd cow
      • svn propset svn:ignore "WEB-INF" web-app/
      • svn propset svn:ignore "core" plugins/
      • svn rm web-app/WEB-INF
      • svn rm plugins/core
      • svn commit -m "Fixing SVN"
      • grails upgrade
        (this re-seeds the non-versioned stuff)
  • sandboxes
    • added sandbox domain class
    • to get default controller, view etc. used:
      • def scaffold = Sandbox in SandboxController
    • validated sandbox filename
  • persistence settings
    • made db persistent for development and production
      • persistent data: edit grails-app/conf/DevelopmentDataSource.groovy to set HSQLDB to "file" and dbCreate to "update"
      • seems to have a 20 second write delay
  • build
    • changed the ant build to cater for the new stuff
    • moved ~/.m2 into sins; now versioning and yam take their gate.jar dependency from sins/maven-repository (so that we can remove the need for installing any maven-related stuff in order to build, e.g. with CC) (see sins/build.xml)
    • upgraded gate jar to 4.0; installed it to local rep using:
      • mvn install:install-file -DgeneratePom=true - -Dmaven.repo.local=/home/hamish/sins/maven-repository - -Dfile=/home/hamish/gate/bin/gate.jar -DgroupId=gate - -DartifactId=gate -Dversion=4.0beta1 -Dpackaging=jar
• [CoW] CoW: Controllable Wiki <HC>
  • general
    • did grails install-templates: creates src/templates
  • serving static pages as a component of dynamic pages.
    • used generate-all to get the controller and view code
    • played with sitemesh api and a view to render a static page
    • investigated the problems with url mappings
      • implemented a UrlRewriteFilter to change url paths to ? syntax
      • see also notes in gatewiki.html
    • upgraded to grails HEAD (0.5.5) and re-test
      • fixed startup .zip problem: delete .tar etc. in grails/dist
• [CoW] CoW: Controllable Wiki <HC>
  • serving static pages
    • changed the views so that links to pages ending in id numbers have a trailing /
    • changed "sandbox" to "wiki" with an action "area" so URLs look like /cow/wiki/area/4?path=...
    • added urlrewrite jar and WEB-INF/web.template.xml to SVN
    • found sitemesh bug on gate.ac.uk index.html and wrote workaround (gate.util.GFiles)
    • using NanoHTTPD.serveFile to construct responses for non-HTML and directory files
      • could in future use java activation to guess mime type and so on; see dani's code, e.g.: def fds = new javax.activation.FileDataSource(fileName);
      • made changes to NanoHTTPD to remove the / from the paths it returns
    • if contentType of response is set to "text/html" grails incorrectly tries to construct a view...
    • finally fixed the paths issues and got static files working properly
      • images now working but not in subdirs because when including the body of a page in a subdir, the relative links will inevitably be broken...
        • tried adding the path part of the ?page= spec, but of course this is not set for the relative links :-(
      • options:
        • go back to the hacked UrlMappingsFilter now the id/ thing has solved part (all?) of the problems that it had? (worked!)
          • clean test in grails
          • implement filter hack and re-test
          • if succeeds try in cow
            • turn off url rewrite filter
            • change cow to use CowUrlMappings again
          • submit JIRA with the patch
        • redirect the *.* urls and use CowUrlMappings to process the dir parts? (rejected)
  • misc
    • installed firefox web dev add-on from xandra: https://addons.mozilla.org/en-US/firefox/addon/60
  • css etc.:
    • modify src/templates before generating controllers and views
    • logo: grails-app/views/layouts/main.gsp link, and put the .png in web-app/images
• [CoW] CoW: Controllable Wiki <HC>
  • cleaned up static code and removed NanoHTTPD dependency
  • serving static pages
    • delete sandbox
    • split WikiController to create WikiAdminController, and move the relevant views
    • add dirListingAllowed to Wiki
    • views: change wiki show to only show id
    • added jetty to sins and checked deployment
    • added new actions and menus, and a top-level, and a short description to Wiki
    • install versioning bean into cow (just add the spring config and make it a member of a controller and it should be autowired) (uses sb mgr for serialisation of access to dir trees)
    • first version of page edit action with YAM summary
  • designed layout and made some examples for the CSS stuff
  • documented the build process
• [CLOnE]
  • Created an ontology lookup component [DD, VT]
  • Created an application integrating a Tokeniser, SentenceSplitter, POSTagger, Morph and the above mentioned ontology lookup component that annotates an input document with mentions of ontological resource (Using flexible gazetteer) [DD, VT]
  • Developed a scoring strategy to combine specificity of the property and also the similarity of its name with existing properties [DD, VT]
• [CoW] CoW: Controllable Wiki <HC>
  • added WikiPage class in WikiController to simplify action logic
  • fix bug with markup in .yams and filling text area for edit (use textarea)
  • added contextPath stuff in YamFile and AbstractTranslator to support translation in a different directory to the rest of the sources
  • finished preview (uses show view with preview set true and temp file POSTed on to save from the save link)
  • finished complete edit/preview/save cycle (save gets a POST with the temp file name that contains the new version of the file, and, in workstation mode, just stuffs it on disk, calls YamFile.translate and redirects to show with a flash message)
  • cleaned up save code and added more checks on WikiPage in WC actions
  • added create page / make directory
• [CoW] CoW: Controllable Wiki <HC>
  • added dir list option to all html show views
  • renamed WikiAdminController as AdminController
• [YAM] Yammering <HC>
  • added gate.yam.dependencies to sins/yam in groovy and get the ant build to add it to the YAM snapshot jar
  • reverted to ant and cleaned up the build.xml

4. Rejected

• File system watching to do updates
  • Watch the registered sandboxes using a low-priority thread that doesn't hog machine resources. The point is to spot YAM files that have changed more recently than the .html generated from them and queue them for regeneration.
  • Consume out-of-date YAM files from the queue and translate them using gate.yam.Yam
• Making YamFile a spring bean: it is hard to do conditional construction of these, and the use case of substituting a different implementation in XML config doesn't apply here.
• Using Tomcat or Jetty virtual directories for serving registered sandboxes. It will need some investigation in Jetty code, and also, the separate impementation for different Web container. Additionally, some implemenations will be essential in order to serve different directories in the same context, while 'context' in a Web container's vocabulary assume 'different application'.
• [CLONE]:
  • Use a listener on the ontology to delete labels at appropriate times (using new ontology API): this is not necessary after removing LabelIndex.
  • RDF labels should not be used for private synonyms. They will screw up NE tagging (discuss with Valy and Kali).
• [YAM] regen the parser html / add the javacc2html jar
  • neither javacc2html nor javatohtml (which also covers .jj and .jjt files) supports java 5 :-(

5. About this Document

This document contains the list of features, functions, technologies, enhancements and bug fixes that would ideally be made part of SAFE. It represents a snapshot of the ever-changing requirements and plan for the system. Items at the top of the list should be detailed enough and fine-grained enough to be implemented. Lower down the list are lower priority features that may or may not be implemented. There are no bad ideas for new features, only low priority features.

This document links to

• user documentation: e.g. the GATEWiki user guide
• developer notes: our design notes

Format:

• [Task type] Task content. (Time estimate.) <People>

The time estimates and people assignment are optional (they're most appropriate for tasks that are near the top of the stack, i.e. will probably be implemented in the coming period).

For example:

• [Core] Remove warnings due to Java 5 Type safety. (2 days) <JS>

When a task is complete move it to the Done section, and link it to relevant user and design documentation.