CoW Backlog

(Maintained by Hamish.)

1. Roadmap (version 1.1, early 2010)

TODO: reprioritise, pull in any pressing stuff from future tasks

• Regeneration conflicts with page editing [AR]
  • It seems that when you make a link to a new page then make the page, then continue editing the linking page (which at this point still has the red link), it results in a conflict, though this can be avoided by refreshing the linking page before continuing to edit so that it does not have a false red link
• upload and newpage flows [IR]
  • upload of an empty file fails without a message
  • uploads and newpage go directly into the sandbox - they should go via staging, at least in server mode.
  • restrict file uploads so users can't upload raw HTML in a location they wouldn't be allowed to view it.
• edit flow [IR]
  • in cases where a previous failed edit is present, and there is also a conflict on update, it seems that edit flow fails to resolve the conflict and therefore prevents saving
• jape yamalyser [HC, NA]
  • fix issues identified in cow/doc/*.yam [NA]
  • do the TODO in PageService.saveEdit [HC]
  • put back the "switch to wysiwyg" in edit.gsp
• when dependencies regeneration is triggered for a file containing includes, the regeneration seems to be run with includes turned off (e.g. edit the download page on g8rs.net, then look at /narrative.html - this now has the includes not processed). [AR]
• bar admin edit on readonly dir (or at least message) [IR]
• search [NA]
  • a query to solr like "tokenisation NOT" generates a server error; it should just message instead
  • prioritise user guide and top-level htmls in solr results
  • group similar pages in solr results
• cache the twitter
• change all .cow:no-iframe and .cow:raw-html to .cow-no-iframe and .cow-raw-html respectively throughout all our repositories
  • note this can only be done once the main gate.ac.uk CoW instance has been redeployed to pick up the required code change

2. Future Tasks

• we should rethink how we preview non-YAM HTML files. At the moment we inject the head section into the preview page which means we can end up replacing values for base target and href which cause weird security problems
• the DB entry for the controller "contact" should be created as part of the JSec stuff in bootstrap, and anonymous given complete access not just read otherwise they won't be able to access the send action
• edit of /gate/doc/index.html throws some exceptions to the log (to do with illegal state when writing the images to the response)
• allow serving of robots.txt on password-protected sites
• cut out most of the stuff from the help area; we can just point to gate.ac.uk/gatewiki now
• edits of non-yam html delete META tags... options: fix the implementation of putBody in FileUtils and go back a non-full-page fck edit; upgrade to more recent fckeditor; try YUI rich editor. while we're at it try to get rid of PageService.FckEscapesFix
• HtmlTranslator excludes e.g. /g8/... from being a candidate missing page; the list of patterns should be a config option in cowrc.groovy that gets pushed into a YamFile static at bootstrap
• svn up operations should trap the file names that are updated and push them to the solr reindexing queue
• quartz jobs and perhaps also webflows should log their start/end times, and put something characteristic in all their log messages; then we'll be able to grep individual jobs/flows out of the log
• what happens to footnotes (or twitters) in included files? [HC]
• g8rs.net
  • get cron to publish security-related log reports somewhere (in the wiki space?) [IR]
• at bootstrap allow raw html in all of the help area (which must be readonly for non-administrators)
• put some navigation in the help area
• is _pageBody.gsp actually in use?
• can we replace helpdocs.zip with a zip of the help svnrep?
• email confirmation plugin
• add tooltips, e.g.: <g:javascript> new Tip( 'newpageName', '<g:message code="cow.new.page.title"/>', {title : '${message(code: 'cow.new.page.tooltip').encodeAsJavaScript()}'}); </g:javascript>
• if you put a link in an existing page to a new (non-existant) page in a subdirectory, you should get the same error message that you get if you try to do this via the new page form, but this doesn't work
• create a new wiki area with a non-existant directory as the target sandbox; the exception that gets thrown by SandboxManager is reported (though not nicely), but the wiki area still gets created. need to trap the exception, report it (and log it) and then abort the area creation. this probably also kills bootstrap when the SbManager call is being made during startup
• navigation menus: to support tablets (e.g. Valy's Nokia) a click on the area of a submenu parent (i.e. the menu item that has a submenu) as opposed to the link on the parent should open the submenu, not the parent (alternatively the arrow area should open the submenu)... difficult: perhaps buy valy a new phone?! or WAP-style tailoring of CSS to different devices, and while we're at it add skins?!
• directories: an anonymous has permissions to browse directories (at the moment, via ?dir, as the link is disabled). we could implement permissions for directory browsing, implementing a specific controller and/or action.

• add exclusions for / and /help to HtmlTranslator so that links to these areas are not considered to be missing pages [HC]
• add %center(...) to YAM
• team wiki ... [MG, HC]
• move cowrc.groovy inside cowrc.d? One less thing to reconfigure when you want to run more than one CoW on the same machine.
• Set the session timeout. See http://www.nabble.com/Jsecurity-setting-the-session-time-out-td20521251.html and any changes to plugin since it was written
• add ability to do syntax highlighting on code listings, with something like %code(...) using http://code.google.com/p/google-code-prettify/
• regeneration
  • setContext on the YamFile
  • use the PageService edit-related methods for regeneration (have a fake user and change PS to allow parameterisation of this)
  • if there's a conflict at commit time, delete and do nothing (cancel this regen; a real person did it...)
  • change PS to delete dependent HTML files before doing update; this will solve potential conflicts in HTML when YAM changes, for example
  • turn quartz job back on and check for conflicts problems

• grails 1.2
  • figure out how to get e.g. the gau site plugin to reference plugin files from the layout without the full path? (cf. the images links in cowguest)
• [CoW] dependencies
  • at the moment, dependencies regeneration runs in the foreground. But should it be a background job? background would be good e.g. overnight, but foreground might be better usually, as it is always run by an admin, who perhaps wants to know when it goes wrong? think about it.
• [CoW] paths
  • create a new install, login as admin:
    • gives: "About to create a new page (or directory) named cow... ok?"
    • but going in via admin browse wikis works ok
    • or, it sometimes gives a directory listing instead of the existing index.html
• [CoW] wiki functions
  • PS refactoring, and gate.util.Yammer
    • PageService could be more concise and clearer if it had an object model for the various types of page and the files on disk (gate.cow.Page, YamPage/TxtPage/BareHtmlPage/Directory extends Page, with overwrite/toStaging/fromStaging methods). PS.analyse would then put the appropriate type into the page model

• [YAM] a snippets plugin? for example, we might want to add a sourceforge mail search plugin... but this shares a lot with teh google plugin; shall we just go on adding more and more plugins to do small jobs like this, or could we allow CoW administrators to create snippets somewhere and then have a general plugin (like the Grails blurb plugin)? disadvantage: ties YAM to CoW, which is not currently the case...
  • <FORM method="GET" action="">http://sourceforge.net/search/">
  • Search <a href="mail/index.html">the mailing list</a>:
  • <input type="hidden" name="type_of_search" value="mlists">
  • <input type="hidden" name="forum_id" value="47765">
  • <input type="hidden" name="group_id" value="143829">
  • <input type="text" cols="30" name="words">
  • <input type="submit" name="Submit" value="Submit">
  • </FORM>

• [CoW] misc
  • autosuggest LinkedOpenData references during authoring?
  • using cow:pageLink etc.: this patch makes the pageLink taglib use the cow:namespace, but seems to break the normalExit state in newpageFlow; no idea why; perhaps try later when at grails HEAD again, and send bug report. in the meantime going with g:page...
  • docs stuff on tropical tolerance to cow docs, update sf.net http://tropicallytolerant.blogspot.com/ http://www.spectrum.ieee.org/print/1699
  • RSS: try the feeds plugin
  • add latex import and support lyx
  • exploit gant latex build for cow latex output
    • includeTool << gate.tools.LaTeX
  • look at S5 for slides output
    • http://www.g2s5.com/presentation/s5express http://www.odelia-technologies.com/g2s5
  • bibtex support via http://jabref.sourceforge.net/ ?
  • alternative approach to exception handling: http://blog.bruary.net/2008/03/grails-custom-exception-handling.html
  • round trip with freemind: http://freemind.sourceforge.net/
  • ability to link between areas, and link to help from doc/main-index-template.yam, for example
• [CoW] semantics integration
  • clone/questio
    • start from olaf's grails-gate-plugin code
    • make questio a grails plugin
    • (maybe) allow yam language to pull in configured predicates
    • make url mappings accept plugin controllers
  • %questio and %sparql (or %owl-nl and %owl-sparql?) in YAM; translation puts AJAX code that pings OWLService <HC, AR>
  • JS code to embed in cow (gatewiki.sf.net) pages that
    • contains a SERQL (or other) query
    • registers an on-load event
    • when the page is viewed in a browser, send the query to the Grails service and renders the result (inserts it into the DOM)
• [CoW] PageController <HC>
  • possible optimisations of the PageService check-out-to-staging code:
    • use the SVNRep API, see: http://www.nabble.com/Re%3A-check-out-a-single-file--p15719277.html
• [CoW] pre-publication area
  • /trunk/ and /branches/ and pre-publication edits and so on
    • create a single "editing" branch. or a new branch for every edit? every user?
    • a new branch can just be a new Wiki
    • new roles: production-editor vs. branch-editor? (former can edit live site)

• [CoW] security, authorisation and authentication
  • should the user's edit profile form require a password entry?
  • when user details are edited via the edit profile form they do not change in the interface until next login. Things like username on the interface etc come from org.jsecurity.subject.Subject.getPrincipal(). This doesn't change until the next login. We can't automatically log out and back in, as we do not know the password, only the hash. we need to somehow force the principal to change. Maybe you can set it? Maybe a question for the list / support?
  • Logging on SecurityFilter: have never managed to get logging working on this. See mailing list question http://grails.markmail.org/message/wqf7m7bymq6tx6f7?q=grails%2Eapp%2Efilter and the refernced JIRA, http://jira.codehaus.org/browse/GRAILS-3155
  • on the mailing list: http://tinyurl.com/b9j4xh The principal tag in the JSecurity plugin isn't HTML encoded and therefore vulnerable to XSS attacks. A patch would be to call encodeAsHTML on the result in the JsecTagLib. This has been done in our own code base, the cow:principal tag, and the question asked again of the mailing list: check there again. Filed a jira: See jira http://jira.codehaus.org/browse/GRAILSPLUGINS-657 Once it has been fixed in the plugin, we can replace cow:principal with jsec:principal
  • protect JsecActionList that are set in configuration from deletion, and warn about deletion of other actions sets if they are in use. This can be done in the same way as for JsecControllerName and JsecRole
  • Directories in the permission model are being changed to be a regular expressions (one for included directories and one for excluded directories). This could be changed to (a) allow a list of regular expressions on each permission and (b) use the easier ant syntax, rather than java regexp syntax.
  • We need to look at the SecurityService.groovy code to remove any redundant code. Also we need to merge the three methods isPermitted, anonAllowed and checkForPermission some way.

• [GATE.ac.uk] "what we do page" <DD,VT> (or: MIMIR search):
  • A JSP that displays Di/Wim's taxonomy and allows users to select one or more concepts
  • When they hit "search" these concepts then become a keyword search against publications.dcs and the results are displayed (list of publications relevant to the set of concepts)
  • http://www.google.com/coop/cse/overview for site search?

• [YAM] Yammering <HC>
  • yam lists in table cells bug (or maybe just an irritant that needs documenting)
    • as for example in sam/doc/sheffield-visit-10-08/future.yam if the first item in a table cell is a list, it has to be preceded by a linebreak, otherwise it causes syntax errors
    • note that the syntax errors can be quite wierd: they include the list item problem, but also "EOF at ..." errors that aren't obviously related to the list (though they go away if you fix the list)
    • it would seem possible to get pushBackListStart to put add a sep token into the stream before the token being pushed back, or similar...?
  • finish fixing latex generation
    • uncomment tex extension in YamTranslatorTest.testHtmlToYamNoIncludes and fix remaining errors
  • %clone
  • slidy (sale/talks/gslidy)
    • should be done like PDF and PDF dropped
    • yam layer
    • doc for team
  • Version 5 bugs:
    • the verbatim mode for HTML doesn't deal properly with angle brackets (which need to be escaped). this is due to html pre not being like latex verbatim. perhaps add another verbatim mode (%<< ?) or make chevrons TargetControls in the lexer?
      • is this now fixed with the html escaping code?
    • commas and citations within footnotes will prematurely end the footnote, and any markup within footnotes will not be interpreted (see also next point), though \, is a valid workaround
      • one option is to have a member predParser and a JAVACODE production that reads it in, counting brackets, then parses it and merges into the tree
      • perhaps a better one is to make a new token "%footnote(" and allow Text() inside it, and mandate escaping of brackets?
      • also, converter xsl written on the assumption that there is no markup in the footnote. may need to be rewritten if this changes (see footnotes in html generation above)
    • (related to previous) left round brackets in URLs cause truncation of the url; we need to allow escaping or quoting (perhaps the same mechanism for all predicates?)
    • includes get wrapped in paragraphs, which results in empty tags in the html. solution: make include different from Predicate and put it under Unit
    • when an image alt tag specifier is enclosed in quotation marks the generated tag puts the attribute quotes in the wrong place
    • spaces at the end of table rows (after the final |) gets discarded
    • if %contents is used multiple times, only the last one works
    • profiling: (yourkit profiler?)
      • check if 1MB docs and above work (double yam-huge)
      • check if parse errors in documents above around 500k (e.g. yam-huge.yam) cause out of memory errors
      • If context isn't set on yams, and generate done withiut links, conversion of the site takes a few seconds. With context set, it takes nearly 3 minutes. (not such a problem for conversion, but an issue elsewhere?)
    • because blockquotes (and things like bold, italic) are just treated as words, they can cause overlap problems (see yam-wierd line 89 and after for example); this would be hard to fix - the workaround is to e.g. make sure to separate the blockquote markup from other constructs that it may enclose
    • an empty URL is illegal: %()
    • all list items, including empty ones, must have a space after the initial "-" or "o". (this is not a bug in fact.)
    • verbatim mode terminates lists; verbatim within lists doesn't correctly place the pre tags in the output and doesn't escape HTML properly
    • when including a file it should be possible to have the title as a higher level heading than the other headings (or simply to exclude the title)
    • if the last thing in a table is a verbatim (%<...>%) item, that thing will not be verbatimised [need to check if this is in the list of known yam bugs below nad if not add it]
    • images in links don't work, e.g.: %(tmp/summer.html, %image(tmp/Screenshot.png) ), and predicate arguments are also not parsed [HC]
  • Version 5 wishlist:
    • something like latex's ref command
    • definition lists
    • appendices
    • captions for figures and tables

• svn replication links: http://www.darcs.net/DarcsWiki/Tailor http://www.wandisco.com/

• [CoW] security on searching
  • At the moment, the search controller is a special case in the SecurityFilters. If someone is trying to access it, then the SecurityFilters checks for permissions on the page controller for the read action (if a user can read a page, then they can search it)
  • We could generalise this special case by allowing permissions to be defined over sets of controllers, in the same way we do for actions:
  • CowPermission(Wiki 1, directory "", controller [page, search], actions [Read])
• [CoW] nutch-solr
  • Upgrade to nutch version 1.0 failed due to some hadoop configuration problem. The problem occurs when trying to upload a huge amount of crawled data to the solr webapp.

3. Completed Tasks

A list of completed tasks is here.

4. About this Document

This document contains the list of features, functions, technologies, enhancements and bug fixes that would ideally be made part of CoW. It represents a snapshot of the ever-changing requirements and plan for the system. Items at the top of the list should be detailed enough and fine-grained enough to be implemented. Lower down the list are lower priority features that may or may not be implemented. There are no bad ideas for new features, only low priority features.

This document links to

• user documentation: e.g. the GATEWiki user guide
• developer notes: our design notes

Format:

• [Task type] Task content. (Time estimate.) <People>

The time estimates and people assignment are optional (they're most appropriate for tasks that are near the top of the stack, i.e. will probably be implemented in the coming period).

For example:

• [Core] Remove warnings due to Java 5 Type safety. (2 days) <JS>

When a task is complete move it to the Done section, and link it to relevant user and design documentation.