Summary

  • CellML’s first curation level checked agreement with a paper; the second checked completeness, consistency and reproduction of its results.
  • Correcting published errors could make a model satisfy the second test while failing the first, a conflict the repository documented in 2008.

A gold star is supposed to simplify matters. CellML managed to give it a philosophical problem. In its 2008 model repository, one star meant that the mathematics matched the published paper. A second level required checks that included reproducing the paper’s results. When the paper contained an error, those ambitions could pull in opposite directions.

The repository’s own account was admirably frank. An early version might follow the publication, errors included. A corrected version could then reproduce the results while breaking the rules for fidelity to the printed mathematics. Two yellow stars could satisfy level two without satisfying level one. The grading ladder had acquired a missing rung.

The paper had to run

CellML described biological models in a form software could exchange and execute. Its online collection had grown from examples of the language into a library of models reconstructed from published literature. At first, curators wrote them by hand and checked that they looked like the mathematics in the paper. Later tools made it possible to ask a less polite question: would this thing actually work?

Catherine Lloyd described that change in her curation presentation at the 26 March 2008 workshop in Auckland. Programs including PCEnv and COR could run models and help inspect them. COR displayed mathematical expressions in readable form and reported problems such as missing information or redundant definitions. Curators then compared simulated output with the published figures; the repository described that comparison as qualitative.

Getting a model to run could become correspondence work. Lloyd’s slides reproduced replies from authors acknowledging missing parameters, a punctuation error in a numerical value, incorrect equation terms and swapped units. The repository staff requested explanations and, where possible, the original code. A download button at the front of the website concealed a rather busy editorial desk behind it.

A successful file also depended on its interpreter. The 2008 documentation described valid CellML models whose equations exceeded what the available simulation tools could solve. The repository therefore also recorded confidence for particular simulators. Those ratings asked whether a model ran in that environment and how closely its output matched the published result.

Which version earned the star?

Revisions received version numbers, while the main model listing opened the newest version by default. That made an old assessment and a newly edited file a consequential pairing. The July 2008 account identified a concrete weak spot: some CellML 1.1 models sat in a separate folder because the existing repository structure could not properly accommodate them. Editing there could leave the curation status behind. The proposed temporary arrangement was to leave assessed uploads unchanged and preserve paths already cited in papers.

Even the third star contained unfinished business. The stated ideal was a domain expert checking physical constraints. In practice, the project also described awarding that level when an original author took responsibility for curation; independent expert review remained a further goal.

The website had made scientific publication answerable to a second reader: software. A corrected equation, an executable version and a recorded test could travel together. The stars were trying to tell that whole story in the space normally reserved for a hotel review.