# Formal methods for Evaluation of Interoperability & Maintainability?

**Category:** [Technical (archive)](https://discourse.openehr.org/c/technical-archive/156)
**Created:** 2008-02-06 16:16 UTC
**Views:** 3
**Replies:** 13
**URL:** https://discourse.openehr.org/t/formal-methods-for-evaluation-of-interoperability-maintainability/14740

---

## Post #1 by @Koray_Atalag

Hi,

I want to learn how we can formally/objectively prove that Archetype
based dual level development formalism alleviates problems of
interoperability and maintainability\. I was wondering if someone did or
know of any such study which applies formal validation methods?

Best regards,

Koray Atalag, MD, Ph\.D\.

---

## Post #2 by @system

Hi Koray

I think we will have to come up with some metrics that are relevant as it has not been done before in the domain space. Clearly modelling at two levels is a common approach - relational databases model the idea of tables with rows and columns, linking keys, data types and indexes. The domain information is expressed in terms of these rows and columns. Many systems driven on metadata do the same thing. What is new about openEHR is a generic approach to allow any base model to be constrained through the use of ADL. The result is that the base model can reflect the general business rules and the fixed information constructs - the archetypes the domain knowledge and how it is represented in terms of the base model. The approach relies only on getting sufficient expressivity at the base level to make the split efficient and safe.

The comparison in health care at present is with HL7 version 3. This has a base model (RIM) from which a new model, an RMIM, is constructed (level 2). The difference is that RMIMs are constructed with alterations to the RIM classes (which are renamed). So we now have a new class based on a pattern. The semantics of the RMIM is a mixture of RIM and RMIM and difficult to untangle. CDA is using templates in the same way as openEHR uses archetypes - to express some domain content. As CDA is already committed to XML, the means of further constraint is limited - hence the use of schematron and other devices.

I guess the first metric that we could consider is the speed at which domain concepts can be modelled and the level of human intervention for documentation and maintenance. The UK NHS, which has the most experience of both, has found openEHR far more efficient to use than MIF template constraints on HL7 CDA. Vendors are cautious and have little experience of openEHR directly as yet.

Clearly archetypes are of great use in systems that use the openEHR Framework and allow use of operability constraints out of the box. What about other vendor systems? Well, Ocean tools are being used to produce inputs for vendors which are formal specifications of data to be stored and communicated. The ability to reuse these artefacts for many purposes - queries, transformations, display and data entry provides another metric that is of use.

We will need some large systems built on openEHR and traditional approaches to compare in the future. For the moment, just having clinical specifications that are computable is the main influence on choosing openEHR - or starting from scratch as new vendors see the benefits (or not).

Cheers, Sam

Koray Atalag wrote:


[details="(attachments)"]

![OceanC\_small.png|74x72](upload://5I367QG2SMJUp18Pt3jF6yz13Ey.png)

[/details]

---

## Post #3 by @gordon.tomes

`Like Koray, I too would like to know`

`*" . . if someone did or knows of any such study which applies formal validation methods? . . "*`

`Regards`
`Gordon`

Gordon Tomes | Acute Care Division | Department of Health and Ageing (MDP 63) | PO Box 9848, Canberra ACT 2601 | Ph 02 6289 5081 | Fax 02 6289 7630 |

<table>
**Sam Heard &lt;sam.heard@oceaninformatics.com&gt;**
Sent by: openehr-technical-bounces@openehr.org

07/02/2008 07:28 AM<table>
<tr>
<td>

Please respond to
For openEHR technical discussions &lt;openehr-technical@openehr.org&gt;

</td>
</tr>
</table>

<table>
<tr>
<td>

To

</td>
<td>

For openEHR technical discussions &lt;openehr-technical@openehr.org&gt;

</td>
</tr>
<tr>
<td>

cc

</td>
<td>

</td>
</tr>
<tr>
<td>

Subject

</td>
<td>

Re: Formal methods for Evaluation of Interoperability &amp; Maintainability? [No Protective Marking]

</td>
</tr>
</table>
<table>
<tr>
<td>

</td>
<td>

</td>
</tr>
</table>
</table>

>  > <br><br> |
> - | - |

Hi Koray

I think we will have to come up with some metrics that are relevant as it has not been done before in the domain space. Clearly modelling at two levels is a common approach - relational databases model the idea of tables with rows and columns, linking keys, data types and indexes. The domain information is expressed in terms of these rows and columns. Many systems driven on metadata do the same thing. What is new about openEHR is a generic approach to allow any base model to be constrained through the use of ADL. The result is that the base model can reflect the general business rules and the fixed information constructs - the archetypes the domain knowledge and how it is represented in terms of the base model. The approach relies only on getting sufficient expressivity at the base level to make the split efficient and safe.

The comparison in health care at present is with HL7 version 3. This has a base model (RIM) from which a new model, an RMIM, is constructed (level 2). The difference is that RMIMs are constructed with alterations to the RIM classes (which are renamed). So we now have a new class based on a pattern. The semantics of the RMIM is a mixture of RIM and RMIM and difficult to untangle. CDA is using templates in the same way as openEHR uses archetypes - to express some domain content. As CDA is already committed to XML, the means of further constraint is limited - hence the use of schematron and other devices.

I guess the first metric that we could consider is the speed at which domain concepts can be modelled and the level of human intervention for documentation and maintenance. The UK NHS, which has the most experience of both, has found openEHR far more efficient to use than MIF template constraints on HL7 CDA. Vendors are cautious and have little experience of openEHR directly as yet.

Clearly archetypes are of great use in systems that use the openEHR Framework and allow use of operability constraints out of the box. What about other vendor systems? Well, Ocean tools are being used to produce inputs for vendors which are formal specifications of data to be stored and communicated. The ability to reuse these artefacts for many purposes - queries, transformations, display and data entry provides another metric that is of use.

We will need some large systems built on openEHR and traditional approaches to compare in the future. For the moment, just having clinical specifications that are computable is the main influence on choosing openEHR - or starting from scratch as new vendors see the benefits (or not).

Cheers, Sam

Koray Atalag wrote:
`Hi,`

`I want to learn how we can formally/objectively prove that Archetype`
`based dual level development formalism alleviates problems of`
`interoperability and maintainability. I was wondering if someone did or`
`know of any such study which applies formal validation methods?`

`Best regards,`

`Koray Atalag, MD, Ph.D.`

`_______________________________________________`
`openEHR-technical mailing list`
[`openEHR-technical@openehr.org`](mailto:openEHR-technical@openehr.org)
[`http://lists.chime.ucl.ac.uk/mailman/listinfo/openehr-technical`](http://lists.chime.ucl.ac.uk/mailman/listinfo/openehr-technical)

``

---

## Post #4 by @system

HI,

Via the Url a PDF/presentation with some calculations.
No message standard, any message standard and the two-level-model paradigm, are compared.
[http://tinyurl.com/26hlch](http://tinyurl.com/26hlch)

Gerard


> Hi Koray
>
> I think we will have to come up with some metrics that are relevant as it has not been done before in the domain space. Clearly modelling at two levels is a common approach - relational databases model the idea of tables with rows and columns, linking keys, data types and indexes. The domain information is expressed in terms of these rows and columns. Many systems driven on metadata do the same thing. What is new about openEHR is a generic approach to allow any base model to be constrained through the use of ADL. The result is that the base model can reflect the general business rules and the fixed information constructs - the archetypes the domain knowledge and how it is represented in terms of the base model. The approach relies only on getting sufficient expressivity at the base level to make the split efficient and safe.
>
> The comparison in health care at present is with HL7 version 3. This has a base model (RIM) from which a new model, an RMIM, is constructed (level 2). The difference is that RMIMs are constructed with alterations to the RIM classes (which are renamed). So we now have a new class based on a pattern. The semantics of the RMIM is a mixture of RIM and RMIM and difficult to untangle. CDA is using templates in the same way as openEHR uses archetypes - to express some domain content. As CDA is already committed to XML, the means of further constraint is limited - hence the use of schematron and other devices.
>
> I guess the first metric that we could consider is the speed at which domain concepts can be modelled and the level of human intervention for documentation and maintenance. The UK NHS, which has the most experience of both, has found openEHR far more efficient to use than MIF template constraints on HL7 CDA. Vendors are cautious and have little experience of openEHR directly as yet.
>
> Clearly archetypes are of great use in systems that use the openEHR Framework and allow use of operability constraints out of the box. What about other vendor systems? Well, Ocean tools are being used to produce inputs for vendors which are formal specifications of data to be stored and communicated. The ability to reuse these artefacts for many purposes - queries, transformations, display and data entry provides another metric that is of use.
>
> We will need some large systems built on openEHR and traditional approaches to compare in the future. For the moment, just having clinical specifications that are computable is the main influence on choosing openEHR - or starting from scratch as new vendors see the benefits (or not).
>
> Cheers, Sam
>
> Koray Atalag wrote:
>
> > ```
> > Hi,
> >
> > I want to learn how we can formally/objectively prove that Archetype
> > based dual level development formalism alleviates problems of
> > interoperability and maintainability. I was wondering if someone did or
> > know of any such study which applies formal validation methods?
> >
> > Best regards,
> >
> > Koray Atalag, MD, Ph.D.
> >
> > _______________________________________________
> > openEHR-technical mailing list
> > [openEHR-technical@openehr.org](mailto:openEHR-technical@openehr.org)
> > [http://lists.chime.ucl.ac.uk/mailman/listinfo/openehr-technical](http://lists.chime.ucl.ac.uk/mailman/listinfo/openehr-technical)
> >
> >
> > ```
>
> _______________________________________________
> openEHR-technical mailing list
> [openEHR-technical@openehr.org](mailto:openEHR-technical@openehr.org)
> [http://lists.chime.ucl.ac.uk/mailman/listinfo/openehr-technical](http://lists.chime.ucl.ac.uk/mailman/listinfo/openehr-technical)

-- <private> --
Gerard Freriks, MD
Huigsloterdijk 378
2158 LR Buitenkaag
The Netherlands

T: +31 252544896
M: +31 620347088
E: [gfrer@luna.nl](mailto:gfrer@luna.nl)

Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

## Post #5 by @Koray_Atalag

Hi Gerard, a very useful document indeed\.\.\.The approach is quite
interesting and solid; no questions mathematically \(at least in my MD
mind\!\)\. I was thinking about brainstorming about finding some metrics
\(logical and feasible to experiment\) to test those issues\. Such as:

Maintenance: comparison of lines of code during maintenance, frequency
of support requests and time to fulfill them, user satisfaction surveys,
cost figures and so on for maintenance

Interop: your points \(i\.e\. \# of interfaces to be implemented, \# of
messages and schemas\), number of transactions, reused fragments, number
of hops during a shared care event \(i\.e\. how many systems particular
data \(EHR extract?\) travels, how many users access it and how\.\.\.\.\.

These are just initial thoughts and I am sure there are already better
ones out there\. I think, seriously, such studies would be very
beneficial for community in convincing interested parties\.

\-koray

Gerard Freriks wrote:

---

## Post #6 by @system

Dear Koray,

A metric from real practice:
- Porting one application from its original database to Ocean Informatics EhrGate took two weeks.
Including the production of a SOAP and .Com interface of the interface of Oceans product.

Problems:
- How do you put figures to the fact that no longer data base conversions are needed?
- How do you put figures to the fact that no longer data is lost because of this?
- How do you put figures to the fact that without reprogramming Healthcare Providers are able to define themselves what data and information they have to store, retrieve, present and exchange and that
they do not need the help of the EHR-system vendor?
- How do you put figures to the fact that vendor lock-in is no longer an issue?
- How do you put figures to the fact that since products based on openEHR/Ocean are a generic tool instead of a **proprietary** product customized for a specific enterprise or department at great cost?
- How do you put figures to the fact that systems based on openEHR/Ocean enable flexibly all ever changing work processes thereby facilitating innovation and market competition?
- How do you put figures to the fact that systems based on openEHR/Ocean never enforces all users to use one set of messages based on one standardized business process?

Gerard


> Hi Gerard, a very useful document indeed...The approach is quite
> interesting and solid; no questions mathematically (at least in my MD
> mind!). I was thinking about brainstorming about finding some metrics
> (logical and feasible to experiment) to test those issues. Such as:
>
> Maintenance: comparison of lines of code during maintenance, frequency
> of support requests and time to fulfill them, user satisfaction surveys,
> cost figures and so on for maintenance
>
> Interop: your points (i.e. # of interfaces to be implemented, # of
> messages and schemas), number of transactions, reused fragments, number
> of hops during a shared care event (i.e. how many systems particular
> data (EHR extract?) travels, how many users access it and how.....
>
> These are just initial thoughts and I am sure there are already better
> ones out there. I think, seriously, such studies would be very
> beneficial for community in convincing interested parties.

-- <private> --
Gerard Freriks, MD
Huigsloterdijk 378
2158 LR Buitenkaag
The Netherlands

T: +31 252544896
M: +31 620347088
E: [gfrer@luna.nl](mailto:gfrer@luna.nl)

Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

## Post #7 by @Juanita_Fernando

Hiya,
I&#39;m thinking of doing some post doc work in this area later on this
year\. I thought you might find this reference useful too Koray:

Smith B, Ceusters W\. HL7 RIM: An incoherent standard\. Studies in Health
Technology and Informatics\. 2006 August 2006\(124\):133\-8\.

Cheers

Juanita

---

## Post #8 by @Koray_Atalag

Hi Juanita and others,

It would be a great research topic and I think one that is needed very
badly from openEHR community\. If I manage to find an appropriate
research position, this would definitely fall within scope of my
research as I already have necessary experience and data in endoscopy as
explained in my thesis\.  I have been  investigating this subject because
of a paper in progress which summarizes my thesis work and I want to
inform readers about other studies which claim Archetype bases two\-level
modelling is superior to classical one in terms of maintainability,
interoperability and domain knowledge governance; preferably with
objective formal methods\. Of course it is hard considering that this is
a new paradigm and tricky due to the nature of problem\. What I saw is
this: formal methods are negligibly scarce and current data is mostly
coming from expert opinion\. There is a very interesting whitepaper
\(2004\) which explains why single level modelling fails in development
and maintenance\. It is not really very scientific\(?\) but you may find it
useful anyways:

A Practical Implementation of a Two Level Archetype Based Clinical Model
http://www.meridianhi.com/IDME_Whitepaper.htm

One last thing about HL7: I read that paper by Ceusters &amp; Smith; it is
interesting though but there is another paper as response from HL7
rounds and both seem to tell about facts from different perspectives\. I
feel that HL7 is over\-criticized here and that this would not increase
the value of this work for sure\. I used v2\.4 messages myself and I found
it very useful like many\. Simply their move with v3 to become a content
standard apart from messaging which is then extended to be an EHR
standard is not an elegant approach\.  Maybe we all criticize about this
aspect, but then it results in a general dislike about whole HL7\. And
keep in mind that only time will show who will survive; think about
\(annoying\) existence of cockroaches appearing many million years before
elegant species in biologic evolution :D

Best regards,

\-koray

Juanita Fernando wrote:

---

## Post #9 by @Juanita_Fernando

Hiya Koray,

It sounds like we may be able to collaborate in the future, which is
fabulous\. I&#39;ll be in touch

Cheers

Juanita

Koray Atalag wrote:

---

## Post #10 by @Koray_Atalag

Hi Juanita, I would be honoured indeed :\)

Just a small remark I want to share with you all: After working in the
field of clinical information systems \(My own firm in Turkey established
in 1996\) I faced with the many problems we discuss here from firsthand
and said enough, sold my shares and got back to academia\. In all the
projects and tenders we got, we in fact lost money due to changing
requirements and a general lack of understanding in procurers and laws\.
I evaluated openSDE, Protege and some other propriety tools but then
discovered openEHR in 2001 or beginning of 2002\. I also did a very risky
thing and based my Ph\.D\. research on this\! Well, it took 7 years for me
to finish it \(\!\) I am happy now that I chose it and very anxious to
conduct some quality research\.

Friendly regards,

\-koray

Juanita Fernando wrote:

---

## Post #11 by @Koray_Atalag

Dear All,

I started this thread to get some feedback for finding methods/metrics
to test &amp; validate maintainability and interoperability \(of Archetype
based two\-level apps\)\. And I got very nice ones; however for
interoperability, apart from Gerard&#39;s interface numbers I did not get
any and interestingly from a quick literature survey I got very little\.
I mean there are some indirect approaches but not straightforward\. My
case is a little more easier:

1\) There is an up and running clinical IS developed with single level
methodology based on an internationally agreed terminology including
relationships and structure \(domain knowledge let&#39;s say\)
2\) There is a complete Archetype model of this terminology using openEHR
RM which can comfortably be considered as a domain ontology \(it has more
than what is given in terminology; i\.e\. existences, cardinalities\)
3\) These two can be said to have the same domain knowledge; ie\. one
hardcoded and one two\-level modelled\.

Now can you think about a method to evaluate the interoperability
levels/score of two systems?

Do we need a remote system for benchmarking \(i\.e\. connect and see how
they interoperate\)?

Sorry to bother\.\.\.\.but if we can get this straight perhaps we can
express comfortably that a two\-level app beats a single level app 7x in
maintainability and 5x interoperability\. Or beats 2x HL7 system in
maintenance but is beaten 2x in interoperablity\.  Perhaps I am being too
naive but it is worth trying\.

Koray Atalag wrote:

---

## Post #12 by @system

Koray,

What metrics do you want to define?

Gerard


> Dear All,
>
> I started this thread to get some feedback for finding methods/metrics
> to test & validate maintainability and interoperability (of Archetype
> based two-level apps). And I got very nice ones; however for
> interoperability, apart from Gerard's interface numbers I did not get
> any and interestingly from a quick literature survey I got very little.
> I mean there are some indirect approaches but not straightforward. My
> case is a little more easier:
>
> 1) There is an up and running clinical IS developed with single level
> methodology based on an internationally agreed terminology including
> relationships and structure (domain knowledge let's say)
> 2) There is a complete Archetype model of this terminology using openEHR
> RM which can comfortably be considered as a domain ontology (it has more
> than what is given in terminology; i.e. existences, cardinalities)
> 3) These two can be said to have the same domain knowledge; ie. one
> hardcoded and one two-level modelled.
>
> Now can you think about a method to evaluate the interoperability
> levels/score of two systems?
>
> Do we need a remote system for benchmarking (i.e. connect and see how
> they interoperate)?
>
> Sorry to bother....but if we can get this straight perhaps we can
> express comfortably that a two-level app beats a single level app 7x in
> maintainability and 5x interoperability. Or beats 2x HL7 system in
> maintenance but is beaten 2x in interoperablity. Perhaps I am being too
> naive but it is worth trying.

-- <private> --
Gerard Freriks, MD
Huigsloterdijk 378
2158 LR Buitenkaag
The Netherlands

T: +31 252544896
M: +31 620347088
E: [gfrer@luna.nl](mailto:gfrer@luna.nl)

Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

## Post #13 by @system

Dear Koray,

I am currently doing a PhD research on the topic of EHR Semantic Interoperability. So I am also very interested in formal ways of measuring system interoperability. As you have discovered, I also found relevant literatures are very few. But we could look for similar ones in other sectors.

I think for measuring interoperability one needs to investigate how the system can exchange information with other systems and use exchanged information effectively. In EHR specifically, it will probably mean we need to look into how EHR can work together with other EHR systemes and surrounding systems, e.g Patient Administrative System, Decision Support System and Quality Registries etc.

Regards,
Rong

---

## Post #14 by @Koray_Atalag

Hi Gerard,

I am talking about metrics like the one you had suggested previously: # of interfaces to be implemented to achieve interoperability with no message standard, some message standard and two-level systems. It is clear and easily computable and very objective. Perhaps it is worth studying qualitative part of the story too apart from just # of interfaces.

Sam also suggested the possibility of assessing archetype reuse (I don't know how to measure that though)

And Rong has suggested to explore how EHR systems work together with other EHR and surrounding systems - that is hard to assess but I think the only way to test it.

In Sam and Rong's case we need some metrics which is applicable to both single level and two level apps and then measure accordingly. Now after quite a literature search and reading, considering both maintainability and interoperability are software quality characteristics there is vast amount of material out there; mainly under Software Product Quality Measures or specific on those attributes. Here is an example on maintainability:

· Fix backlog and backlog management index

· Fix response time and fix responsiveness

· Percent delinquent fixes

Fix quality

* backlog management index (BMI)=<img src="file:///C:%5CDOCUME%7E2%5Ckoray%5CLOCALS%7E1%5CTemp%5Cmsohtml1%5C01%5Cclip_image001.gif" v:shapes="_x0000_i1025" height="38" width="374">

I think an archetype based two-level app will beat with this index

* Fix Response Time and Fix Responsiveness: this will be the killer metric I assume.

Reference: Software Quality Metrics Overview, Book Chapter (4) By Stephen H. Kan., Dec 20, 2002

There are many many more of those; and I think we need to identify relevant ones, especially the metrics which forecast on the quality of product based on design, before actual implementation.

Sorry to bother with all this on discussion list and if there is more interest we can continue on the wiki.

-koray

Gerard Freriks wrote:

---


**Canonical:** https://discourse.openehr.org/t/formal-methods-for-evaluation-of-interoperability-maintainability/14740
**Original content:** https://discourse.openehr.org/t/formal-methods-for-evaluation-of-interoperability-maintainability/14740