There was a topic with this title in an internal SEC discussion that @borut.jures pointed would be interesting to share wider, e.g in the light of @rubentalstra’s work with GitHub - rubentalstra/Veredictum: An independent conformance instrument for openEHR CDRs: spec-cited test catalogue, pure-function verdicts, measured performance. The released openEHR specs are the only authority. · GitHub so I start this thread and will move the old posts here.
I have made good progress in creating a reasonable set of documents that are moving toward a v1.0 usable status. The ‘big’ document is the platform test schedule, which consists of test cases lifted from EhrBase (@pablo’s work) plus robot scripts (Wlad’s work), test data sets and other resources.
There is more work to do, e.g. to:
- finalise what a test report looks like
- finalise a conformance certificate
- finish the main test schedule
- make it runnable
However, even as this is going on, I suggest all implementers need to consider ASAP how they will implement their own testing to use this resource, and ideally procurers will think about how they can use it.
Most parties are going to say: how do I download and run this? Well the source files are mostly there, taken from EhrBase, and in theory with some work, the entire repository can be made runnable in a CI environment that looks like the EhrBase one.
In procurement situations that are likely to occur this year and next, the above is a very concrete question.
I think we need to move to a mode where primarily implementers (let’s assume Industry Partners) are working on this together. If there was participation from the procurement side, even better.
How do we get there from where we are now? I think we should potentially try to convene call(s) with as many industry partners as possible to work out next steps.
thoughts from the SEC?
Side question: we have (as agreed) made a copy (not even a technical fork) of EhrBase test documentation, scripts and files. Now both EhrBase and openEHR have an almost identical doc+code base, but of course it will (already is) diverging as EhrBase keeps moving; in future when the openEHR conformance repo is being maintained, we have the standard danger of maintaining two copies of logically the same thing. We don’t really want this. There are some differences:
- the openEHR conformance tests have systematic unique identifiers; EhrBase has ‘heading-based’ ids
- some tests and content in EhrBase is specific to EhrBase, and is not needed in openEHR
However, all the detailed test logic, data sets etc are duplicates - for now. Is there a way for openEHR and EhrBase to improve this situation so that we don’t have a double maintenance problem? @pablo , @birger.haarbrandt - thoughts?
We have a description and an example of the report on 1.3.6. and 4. in the Conformance Framework Design doc https://drive.google.com/file/d/1NKT9Q1eo42DKrcHp7g5Oys6t2dLupcvh/view?usp=sharing
There are two key points:
- Document the result for each test case + data set combination in an individual way
- Have a processable expression of the report (I defined a sample JSON document)
The report has these sections:
- Product identification
- Developer / provider
- Test case + data set result
And there are two report views:
- Summarized view with metrics (mainly for charting)
- Detailed view
I took many ideas from the FHIR community conformance reports:
- https://www.projectcrucible.org/
- https://www.projectcrucible.org/scorecard/58ed404f6372750d0a000000
- https://inferno.healthit.gov/
- https://pyrohealth.net/pyro-r4-fhir-metadata
This is awesome IMHO
This is very important, and there is a need of some cross-coordination. I believe you know the last commit that was done, or at least the date when you pulled the docs + test from the EHRBASE repo, and we can use that as a baseline to track the changes I did to the test case specifications after that, which were mainly the data validation test suite (for composition data) and updates I did this week to the EHR test data sets because I detected some missing cases and added them.
I’m guessing the openEHR repo will keep pulling from the EHRBASE repo since EHRBASE will move faster than the openEHR test cases, which is a realistic context.
A more ideal context would be to maintain everything at the openEHR repo and EHRBASE pulls the conformance test cases from there.
BTW I’m reading the docs and I’m not sure what’s the best way of adding comments on each section (we had this issue with other specs in the past, like the review I did for the AQL spec, I was actually copying parts of the spec to a google doc and commenting there, which is not so practical).
I’ll add one simple comment here: on section 3.1, funtional conformance mentions test case but doesn’t mention data sets, which are required to any test cases related with providing content/payload to the SUT.
Then in content conformance, we call that data validation conformance. It’s just that content is to wide IMO, for instance in the CKM content would be archetypes and templates, for which we should also have a conformance area since we have formats and rules for archetypes and templates that should be met.
In critical attributes, a term I don’t understand, but seems to be related to non-functional properties, I think in the Conformance Framework that is specified as a Conformance Statement, which is more a declaration from the provider than something that could be tested, and the role of that document is to be part of contracts generated from tenders.
Yes, the commit ids are in comments. I took a new copy of _resources and noted the commit id of EhrBase for that as well. So we can find the EhrBase versions no problem.
Unfortunately with the Asciidoc / markdown approach, ‘commenting’ is not like with Word etc - mostly we tend to do it the old way, with refs like ‘section 2.1 para 3’ etc.
Need to fix, with your guidance - it may be that I don’t yet completely understand the organisation of the /doc and /test hierarchies, so feel free to point out what needs to be fixed, or indeed, you can do it via pull request. But we need to agree more widely on the structure of the specifications-CNF repository, which of course is not the same as the EhrBase repo.
That’s a better name, I’ll change that.
'Critical attributes`is a common synonym for ‘non-functional attributes’ or ‘system quality attributes’. I don’t mind which of those terms we use, but it covers non-functional items like (stolen from here):
- Performance – shows the response of the system to performing certain actions for a certain period of time.
- Interoperability is an attribute of the system or part of the system that is responsible for its operation and the transmission of data and its exchange with other external systems.
- Usability is one of the most important attributes, because, unlike in cases with other attributes, users can see directly how well this attribute of the system is worked out.
- Reliability is an attribute of the system responsible for the ability to continue to operate under predefined conditions.
- Availability is part of reliability and is expressed as the ratio of the available system time to the total working time.
- Security is responsible for the ability of the system to reduce the likelihood of malicious or accidental actions as well as the possibility of theft or loss of information.
- Maintainability is the ability of the system to support changes.
- Modifiability determines how many common changes need to be made to the system to make changes to each individual item.
- Testability shows how well the system allows performing tests, according to predefined criteria.
- Scalability is the ability of the system to handle load increases without decreasing performance, or the possibility to rapidly increase the load.
- Reusability is a chance of using a component or system in other components/systems with small or no change.
- Supportability is the ability of the system to provide useful information for identifying and solving problems.
We don’t need many of these in openEHR conformance of course. At some point we might want to specify criteria for security, maintainability and scalability.
Commenting separately on this - perhaps you, Wlad, @birger.haarbrandt can have a think about the identification system I have added (which we could still change) and see if you think it would help in EhrBase - since I have converted 90% of the /doc and /robot files over, you would not need to do any work. Perhaps we should have a call to discuss this, because I’d really like to minimise repeated work for everyone!
I’m still unfamiliar with the term, at least from my university time and from software engineering bibliography.
What I know from ISO/IEC 25010 is the Software Quality Attributes or Characteristics, which kind of match your list: ISO 25010 (previously ISO/IEC 9126 which is the norm I studied at the university in the software engineering courses ISO/IEC 9126 - Wikipedia)
Did a quick search of “critical software attributes” and can’t find any references to the term, while “software quality characteristics” has many results on the first page of google, accessing them I see references to the ISO standards.
IMHO we should stick with international standard terminology.
I think this goes beyond identification of test cases, it’s more about how we manage the documentation, test cases and test data sets evolution. Consider for EHRBASE we are linking documentation, implementation and issues (issues are raised with the test case ID that is on the doc and test case implementation in Robot.
The only thing we don’t have in the EHRBASE environment, and it’s important for the Conformance Verification, is the test data set identification (see 1.3.6 and 4.2. CaboLabs Health Informatics, Standards and Interoperability). This enables reporting results by test case and data set (different combinations of data sets for the same test case will produce different results).
Thinking about all this, for EHRBASE we need to maintain these test cases and data sets in a quick pipeline. I doubt we could have releases in terms of days for the openEHR ones. This is more a question than an opinion 
In section 3.2. openEHR Platform Conformance Test Schedule “For any given conformance point, there may be multiple tests, each of which is individually identified.”
I would add: “for each test case there may be multiple test data sets”.
An prefer to use the term test case instead of just test, because we have:
- test data sets
- test cases
- test suites
(see glossary on the last version of the conformance framework doc CaboLabs Health Informatics, Standards and Interoperability)
We need to individually identify suites, cases, and data sets, since for one combination of test case and data set, we will have one result, which we can then compare with an expected result to state if the operation conforms or not.
I guess section 3.3. that is TBD is related to these 2 docs:
- https://github.com/ehrbase/ehrbase/blob/develop/doc/conformance_testing/COMPOSITION_VALIDATION_STRUCTURE.md
- https://github.com/ehrbase/ehrbase/blob/develop/doc/conformance_testing/COMPOSITION_VALIDATION_DATATYPES.md
I have separated what is structural validation from data value validation. While designing those test suites I have detected three types of validations:
- RM/schema: data/structure doesn’t conform with RM or the corresponding schema for the serialization format used (structural model validation)
- Invariant: extra rules added to the RM to which the data/structure should comply (non-structural model validation)
- Archetype/OPT: metamodel validation
The terms “structural/non-structural” applied to the validation means: if the data breaks anything defined in the UML class diagram, that is structural. Any extra rule to the UML, like in OCL or whatever invariant language, is non-structural. That is referring to the model itself, not to the “structure validation” mentioned above that is related to check types, occurrences, cardinality and existence constraints in the data provided conforms with the constraints defined in the Archetype/OPT, so “structure test suite” is related to the “metamodel validation”.
In section 4., first bullet list, we could have links to the specific interface definitions.
Below in “The artefacts that flow through these interfaces are defined by the following specifications:”, I think more than “flow through”, is really that this spec “depends on” these other specs. So instead of referring to the artifacts, I think it’s more clear to refer to a spec dependency map, maybe with a diagram?
In concrete terms, what flows through these interfaces is the serialized expression of the artifacts defined of specs X, Y, Z.
I know I’m to picky, I understand what you wrote, I’m thinking about someone else reading the specs and being clear for them.
UPDATE: Thinking this spec dependency thing a little bit more, I think the dependency should be with the Service Model, then theSM will have it’s own dependencies with RM and AOM.
For the test data sets and for any examples that appear if the test suite definitions, we can comment which spec defines that data set/example, including links between specs.
Agree. In fact, I added:
For each test case, there may be multiple data sets. A ‘test’ is therefore the execution of a particular test case with a particular data set.
Yep - this is just the overview - and I had not updated that bit, it is done now.
I converted the originals of your COMPOSITION_VALIDATION_STRUCTURE and COMPOSITION_VALIDATION_DATATYPES to the content in this section.
Agree.
I would probably just include this in ‘structural’ because it is rules about generic model-described structures - just expressed separately from the classes. Consider that a cardinality constraint for example is just an invariant constraint on multiplicity of some attributes.
Invariants in the RM can be distinguished from rules found in archetypes or elsewhere, which of course operate a whole level up.
Fixed.
That could be useful, but I would not want to put such diagrams primarily in the conformance schedule - but we could add such a diagram to the SM Platform Spec and refer to it from here. Unfortunately I don’t have such a diagram right now. For now I’ll put a TODO marker in the test spec.
haha, yes…
Agree.
Now, I think we need to work out a data set identification system.
Most of the above changes are up now.
Thaks @erik.sundvall I agree that the Veredictum approach has real merit. What is missing though, I think is some sort of ‘business perspective’ on the result of a conformance run. What does it mean to me as a potential CDR consumer if a CDR fails a particular test? Is this important or relevant, particularly if I do not have immediate complex needs. Is this even a proper openEHR CDR or just a sham(we know people have tried)
We tried to reflect this in our Postman-based tests which regarded Level 0 fails as being so critical as telling the consumer that this fails to perform even basic CDR functionality, whilst Level 1 implied technical but not critical fails such as incorrect REST response codes, headers or the recent issues with missing archetype_details in EHR_STATUS. All Spec fails but none are critical in terms of actual functionality. We also have the issue of different levels of support for different parts of the spec e.g Demorgraphics, and of maturity/ coverage of the spec especially for AQL.
How might that kind of ‘business layer’ work with Veredictum?
@ian.mcnicoll I believe @erik.sundvall wanted to point to our conversation with @thomas.beale from back 2022, when we were actively working in conformance.
I would say, though it’s not nice that I say it for something I build myself but, the spec I have built during my work on EHRbase’s conformance improvements from 2018 to 2022 contemplates most of the areas, from testing to report, and even software certification (which was the final goal), though openEHR at that time didn’t seem interested in that approach (hopefully this will get a second chance!).
Maybe a second read with new eyes might resurrect my work into something that can be peer-checked and incorporated into the specs.
The repo includes:
- A presentation I gave for the SEC from 2019
- The Conformance Verification Framework (it’s a design document with all the requirements, check the last version)
- Testing specification (design of the test cases, following the design document, that can be implemented in any technology)
- Test suites (sample implementation of some of the tests specified in the testing spec. This is what Veredictum could do. I also have tested other implementation technologies.)
- A presentation I gave at the openEHR Conference back form 2023 in the NL (shows how everything works and is much updated than the presentation I gave to the SEC)
UPDATE: I found my presentation for the 2023 conference was already in that repo too.