Skip to main content

Home/ OpenDocument/ Group items tagged testability

Rss Feed Group items tagged

Filter: All | Bookmarks | Topics Simple Middle
Paul Merrell

untitled - 0 views

www.dpawson.co.uk/...090612.html

ODF specificity quality testability

shared by Paul Merrell on 18 Jun 09 - Cached
  • Most (quality) specifications provide clear instructions using those magic words SHALL, SHALL NOT, and MAY where those words have a defined meaning for an implementor. Paragraphs are clearly identified as either normative or informative. That way an implementor knows what they must and may implement to claim conformance against a specification. This approach has been well established over time as a sensible way for spec writers and implementors to work
  • Most (quality) specifications provide clear instructions using those magic words SHALL, SHALL NOT, and MAY where those words have a defined meaning for an implementor. Paragraphs are clearly identified as either normative or informative. That way an implementor knows what they must and may implement to claim conformance against a specification. This approach has been well established over time as a sensible way for spec writers and implementors to work That is the way quality specifications are written. For example, ISO/IEC's JTC 1 Directives (link to PDF) requires that international standards designed for interoperability "specify clearly and unambiguously the conformity requirements that are essential to achieve the interoperability." With that clarity, conformance is testable and can provide confidence of interoperability. A suite of tests may be developed and applied to an implementation to determine which tests pass, which fail, and hence arrive at an objective pronouncement on conformance of an implementation against the entirety of the specification.
  • In a quality specification, it should be feasible to select a normative paragraph, identify a conformance test for it, and make a clear statement that this test proves that an implementation meets (or fails to meet) that requirement. Call it a test plan: define the tests (test specification), define the expected set of results, and define what constitutes a "pass" of each test that establishes conformance. The plan then provides the matrix of test spec against requirement. Simple.
  • ...4 more annotations...
  • Rob Weir of IBM chaired (apology for the misuse of that last word) the formation list and then simply announced what the charter would be rather than seeking consensus among the list participants. As part of this process before that charter was produced and while I still naively believed that consensus was a goal, I sat down with ODF 1.1 and did a paragraph-by-paragraph review for testability. The numbers were quite revealing. I completely reviewed only the first four major sections and found very few clear requirements. The majority were mere statements with no normative language used to identify what was required or optional. Implementors would have to make their own interpretation.
  • It's ironic that the chair viewed as good news the fact that there were far fewer testable paragraphs than he had predicted. But his prediction of 10,000 test cases is probably far closer to how many testable paragraphs there should be; my counts were actually bad news.
  • All of the above leads to the interesting question of just how the chair expects to accomplish much that is useful in regard to ODF conformance testing before the specification is amended to tighten up the language and add clear requirements. The syntax conformity is already handled by validation against the schema, but the semantics are woefully under-specified.
  • Summary: ODF 1.1 isn't verifiable as a specification. From a fairly cursory review of the latest draft, ODF 1.2 will follow the same path. With OASIS now being more demanding regarding conformance requirements on every specification and with ISO/IEC taking a closer interest in liaison with the ODF TC, I find it hard to see how the ODF TC co-chairs can maintain this view toward verification.
1 - 1 of 1
Showing 20▼ items per page