I’ve been working for a while in the CONFORMANCE framework for HiGHmed, and now we are having interest in this area from Solit-Clouds, and having some extended conformance discussions internally. So there are some areas we need to discuss with the community since this will be a CONFORMANCE framework anyone can use to validate their implementations.
The current framework is focused on testing conformance with the Service Model via it’s only standard ITS spec, the REST API.
This framework is composed by a test suite design (ehrbase/doc/conformance_testing at develop · ehrbase/ehrbase · GitHub) and an implementation in Robot Framework (ehrbase/tests/robot at develop · ehrbase/ehrbase · GitHub).
The rational behind those tests is:
- Design should be technology agnostic (it doesn’t mention any specific implementation or even a REST API)
- It is independent from the interchange format (it just mentions the system under test should at least support one openEHR exchange/serialization format)
- Is based on the Service Model (openEHR Platform Service Model), so it defines test cases over each interface and function there.
This opens the door for different discussion topics:
a. Should testing for formats be part of a CONFORMANCE test suite?
b. Should we differentiate levels of CONFORMANCE? for instance, CONFORMANCE with functionality vs. support for exchange formats.
c. Should we differentiate scopes of CONFORMANCE? the format specifications are still part of the standard but are a different level of spec (ITS).
Another thing mentioned today was the requirements for openEHR data validation, that is which data errors are checked by systems receiving openEHR data and how errors are reported back to the source.
Conceptually, I like to divide this into two categories: syntactic (validation against schemas), and semantic (validation against templates). In this context we are thinking mostly of COMPOSITIONs, but this should include FOLDERs, EHR_STATUS, EHR, CONTRIBUTION, PARTY, etc.
There is of course the challenge of having templates for classes that are not COMPOSITION, which is an area we have shelved for a while.
So there could be another area of CONFORMANCE about how data is validated and error reports are sent back to the source.
For instance, developers would want to know if a system is compliant with certain format, but, is that required for checking CONFORMANCE for a procurement?
So is “technical conformance” at the same level of “functional conformance”? Or do we need to separate CONF specs in terms of the needs of the person trying to evaluate an openEHR implementation?
For instance, about procurements, I think functionality CONFORMANCE is key here, the system should provide certain functionalities via one implementation (any) of the Service Model, because that is the spec that defines what an openEHR system should provide.
IMO it’s better to have an abstract / platform independent conformance test suite specification, so the implementation can vary. Today we have Robot, but, if needed, tomorrow this could be implemented in anything, even as an Insomnia or Postman script.
So that alone could work for procurements to express requirements and to check what is actually provided by a solution presented for that procurement.
Though, we need to improve the current SM to use it as a formal definition of what an openEHR system could/should provide (one single system might not implement all the services).
Then, another test suite is required for a technical conformance checks. That would be a set of tests to verify ITS specs. This includes: formats, data validation, validating schemas against BMM, …
From that, another interesting idea: for each openEHR spec component we might need to define a test suite! (to verify systems implementing those, are really implementing things right).
I know this is long but we would like to hear from the community to have more input. I think we got enough experience on this area to move it further, and be closer to say “how compliant” a system is with openEHR. This is also a step closer to formalize procurements, have an official openEHR system certification in place, and improve the required specs to get this right (specially SM and CONF).
What do you think?