Data quality questions/ proposal

Clinical (archive)
1

One of the major requirements we have is what I call a ‘data quality marker’. So the blood pressure recorded is 88/124 but what is the ‘value/ quality’ of this measurement.
IMHO any recorded value is useless unless the quality of this measurement can be established and taken into account when interpreting the data

In order to establish this data quality we need to add some attributes to the observation archetypes used to record such measurement.

So far as we can see now we think that these attributes are a data quality field and a device/instrument reference (which requires a device archetype) and this is what we would like to propose to the community.

Since I don’t know exactly how to do that and we still have many unanswered questions I’ll describe what we’re thinking about. It’s very well possible that these thing are already in place, in that case we’re aren’t aware of that and would like to be pointed in the right direction.

In our ‘model’ data quality can be described as: excellent, good, doubtful and insufficient.

Here the first hurdle arises: one needs a protocol to define what is excellent, good etc. These are probably ‘local’ criteria, so the can’t be embedded in a general archetype.
Our idea is to create a specialisation of the observation archetype in question, in which the local protocol is attached. For instance this blood pressure archetype with the local Dutch data quality criteria would be openEHR-EHR-OBSERVATION.blood_pressure-data_qualityNL.v1.adl

To give an example these are the criteria for blood pressure we’re thinking off:

Excellent:
data measured/obtained by a qualified healthcare provider, with a certified instrument/device that’s calibrated against a ‘golden standard’, the measurement error is within a tight bandwidth (<5%), the validity duration of the calibration isn’t expired, maintained on time and by qualified personal
(This can’t be met when self-measuring in the home situation)

Good:
data measured/obtained by a qualified person (this can also be a properly trained patient/citizen), with a certified (CE marked) instrument/device that’s self calibrating, the measurement error is within a tight bandwidth (I.e. machine is approved by the European society of Hypertension (ESH), the machine isn’t broke and functioning well

Poor/ Doubtfuldata measured/obtained by a qualified person (this can also be a properly trained patient/citizen), with a certified instrument/device that’s self calibrating, the measurement error isn’t within a tight bandwidth (CE marking alone allows measurement errors >7%), the machine isn’t broke and functioning well

Insufficient: in all the other situations

As a consequence we need to add at least one other attribute: a reference/ link to the device used. In our opinion there should be a separate archetype for a device/instrument. In this archetypes not only the unique identifiers of this device are recorded but also information about calibration, maintenance etc. etc. So far as I understand/can see such an archetype doesn’t exist today.
Our idea is to use the demographic archetype model for this. In fact there is already a demographic archetype subtype for ‘agents’. So either we extent this subclass so it can be used for devices or we create a new archetype class for devices/instruments based on this agent model.

Another thing that is already established is the capability of a healthcare professional. I.e. is this person properly trained to operate a device/instrument? In that respect I would like to add similar capabilities for non-health care professionals. In the above case patients/ citizens also can measure their own blood pressure. Before they can do that, they’re trained and examined. Only then they’re capable of producing ‘good quality’ (provided that they meet the other criteria as well) data.

Can anybody please comment on this? As stated before it would be really of great help if we could organise some sort of ‘archetype boot camp’ to create an expanding community of clinicians who know how to create archetypes and harmonize the ‘wishes and ideas’ that will come up as soon as more people start creating and using archetypes.

Cheers,

Stef

2

Hi Stef,

Very interesting and, in principle, quite correct but I am not sure how practicable this would be in the real world. There are so many variables that might determine whether a Blood pressure is either accurate or appropriate, including the purpose for which the historical blood pressure reading is being monitored or reviewed. For instance a series of post-op Blood pressures may be accurately taken but be quite inappropriate to use for long term hypertension monitoring.

The other problem is that you can only make some of these measures of data quality in retrospect e.g. You may have a BP device that has been calibrated correctly but later malfunctions. Or you may have a seemingly competent patient who turns out to have been messing up the process.

I can see some value in a simple flag (defaulting to false) to identify BP readings that should not be used for monitoring purposes because they are known to have quality issues or were taken in inappropriate circumstances e.g post-op or severe illness but I think your data quality markers may be too complex to be workable.

Regards,

Ian

McNicoll Medical Informatics

3

We had many discussions over the years on this topic in openEHR, and eventually decided to get rid of built in flags to do with certainty. There are ways to express certainty (see below) but we need to understand the problem first. This is our current understanding.

  • there is a concept of ‘accuracy’ which is the level of error in instruments, other objective measuring mechanisms. This is modelled in the Quantity data types, see the Data Types IM (http://svn.openehr.org/specification/TAGS/Release-1.0.1/publishing/architecture/rm/data_types_im.pdf) for a detailed discussion. Mathematically accuracy defines a confidence interval into which the value on a dial, readout is in; mechanically it might correspond to some tolerances in manufacture etc. The main thing is that ‘accuracy’ is a concept of objective error. This concept seems to apply pretty much to quantitative values, including ordinals, and is this accommodated in the Quantity package. There is also a magnitude_status attribute which allows ~, <, >, <=, >= to be used, since some lab software generates such markers.

  • there is a concept of ‘confidence’ in a clinical judgement, e.g. being 75% sure about a diagnosis (which is really a differential diagnosis), risk of a problem occurring in a patient etc. This information can be archetyped in an Evaluation archetype, and there is no standard for it - some people use percentages, some use confidence - very low/low/medium/high/very high, there are other schemes, and there have been studies on how terms like high/medium/low correspond to percentages - which show that doctors don’t all equate them to the same numbers.

  • either of these can be adjusted in restrospect, using versioning (built in to openEHR semantics).
    After some 5+ years of debating this issue in a very wide context, including on these lists, I think the current approach is probably about right, but some may not be surprised to know that some of us were just as certain years ago of the need for a ‘flag’ as you are now…but the more you try using it, the less it works. So we have restricted such a flag + accuracy to Quantity types, and otherwise confidence is archetyped.

hope this helps.

  • thomas beale

Ian McNicoll wrote:

4

Hi Stef,

Very interesting and, in principle, quite correct but I am not sure how practicable this would be in the real world. There are so many variables that might determine whether a Blood pressure is either accurate or appropriate, including the purpose for which the historical blood pressure reading is being monitored or reviewed. For instance a series of post-op Blood pressures may be accurately taken but be quite inappropriate to use for long term hypertension monitoring.

I totally agree. That’s why I think that such protocols should be (local) specialisations of a otherwise generic basic archetype. Altough bloodpressure isn’t a ‘critical’ value, in my opinion it’s critical to be able to establish data quality when using data of ‘others’. At least here in the Netherlands, one inherits the responsibility for the use of it, when one chooses to deploy data of poor quality, f.i. in the treatment of a patient. If the patient dies because of this both the provider and the user of the data are held equally responsible. If data can’t be used because of the lack of quality, what’s the use for an EHR?

The other problem is that you can only make some of these measures of data quality in retrospect e.g. You may have a BP device that has been calibrated correctly but later malfunctions. Or you may have a seemingly competent patient who turns out to have been messing up the process.

That’s why we need a seperate device/instrument archetype which records things as calibration en maintainance of the device. From that archetype the ‘actual’ status of the device can be dertermined. We envision apllacations in which a reminder/ warning will show up when a device doesn’t meet the criteria and prevents data of being entered in the system. The beauty of archetypes is that one also can ‘go back in time’. So f.i. in case of a dispute i t’s possible to ‘establish’ the device status at any moment.

I can see some value in a simple flag (defaulting to false) to identify BP readings that should not be used for monitoring purposes because they are known to have quality issues or were taken in inappropriate circumstances e.g post-op or severe illness but I think your data quality markers may be too complex to be workable.

Coming from a semi- pharmaceutical back ground it really surprises me how high quality standards there are for the development of any drug and how these standards largely disappear as soon as health care providers get involved. It looks like the fact that something is done by a doctor is sufficient to prove the quality of it:-)

Cheers,

Stef

5

Dear all,
I hope I am not taking your time with an irrelevant subject, but US
health and Human services has posted a report on the prototype
architectures, and this seemed like a good place to share this info. My
apologies if this is off-topic or a duplicate post.

http://www.hhs.gov/healthit/healthnetwork/resources/summary_report_on_nhin_Prototype_architectures.pdf

Best Regards
Seref Arikan

6

We had many discussions over the years on this topic in openEHR, and eventually decided to get rid of built in flags to do with certainty. There are ways to express certainty (see below) but we need to understand the problem first. This is our current understanding.

  • there is a concept of ‘accuracy’ which is the level of error in instruments, other objective measuring mechanisms. This is modelled in the Quantity data types, see the Data Types IM (http://svn.openehr.org/specification/TAGS/Release-1.0.1/publishing/architecture/rm/data_types_im.pdf) for a detailed discussion. Mathematically accuracy defines a confidence interval into which the value on a dial, readout is in; mechanically it might correspond to some tolerances in manufacture etc. The main thing is that ‘accuracy’ is a concept of objective error. This concept seems to apply pretty much to quantitative values, including ordinals, and is this accommodated in the Quantity package. There is also a magnitude_status attribute which allows ~, <, >, <=, >= to be used, since some lab software generates such markers.

  • there is a concept of ‘confidence’ in a clinical judgement, e.g. being 75% sure about a diagnosis (which is really a differential diagnosis), risk of a problem occurring in a patient etc. This information can be archetyped in an Evaluation archetype, and there is no standard for it - some people use percentages, some use confidence - very low/low/medium/high/very high, there are other schemes, and there have been studies on how terms like high/medium/low correspond to percentages - which show that doctors don’t all equate them to the same numbers.

  • either of these can be adjusted in restrospect, using versioning (built in to openEHR semantics).
    After some 5+ years of debating this issue in a very wide context, including on these lists, I think the current approach is probably about right, but some may not be surprised to know that some of us were just as certain years ago of the need for a ‘flag’ as you are now…but the more you try using it, the less it works. So we have restricted such a flag + accuracy to Quantity types, and otherwise confidence is archetyped.

hope this helps.

I don’t want to be offensive but it doesn’t. This is a probably correct but for me way to technical answer to a practical question.

As a clinician I need to be able to establish that de a certain value that is send to me is reliable. Therefore I need not only to know who measured this data but also which device was used and was this device properly calibrated and maintained at the time of measurement, was the performer properly trained to do the measurement en was the meausurement protocol followed. I know blood pressure isn’t the best example so lets take a more critical one like blood clotting time.

All this I want to check against our ‘internal’ protocol so that I know whether I can use this data or discard it.
This is one of the major reasons (at least in the Netherlands) that medical specialist discard data that already gathered by a GP (everybody has to go to a GP first in the Netherlands. The GP examines a patient and might perform some additional tests/ measurements before he/she decides to refer to a specialist). The other major reason is that data can’t be exchanged between the different systems.. One of the reasons that a EHR is promoted, is that data can be shared and re-used. Especially the latter is of great importance since many decisions makers firmly believe that’s the way to controls the expanding costs of the system. If we can share data thanks to openEHR but still the data quality remains questionable (and yes this has all to do with trust, but also with protecting ones income/ position) people will use that argument to reject the data and remain working the old fashioned way.

What I’m looking for is a practical way to establish this situation. Any suggestions?

Cheers,

Stef

7

Hi Stef

I think we can do some things that will be helpful.
Accuracy

  • As Tom has said we can add some accuracy (which can be a %) to any reading, so it is possible for devices to give a reading with an accuracy. This appears to cover one need that you have - we need to see what is happening with the device standards on this front from the Continua group and see how they propose to deal with this. http://www.continuaalliance.org
    Device and callibration

  • You and others can develop a terminology for home measuring devices - perhaps the Continua group will provide this on asking. This can be linked to a data store of the measuring characteristics of the device. This can be set as a default value within each users template - unlikely to be entered manually every time.

  • You can ask for the date of last callibration of the device (perhaps who did it) - although this is unlikely to be filled in unless it comes as an automatic feed from the device.
    Your idea for an archetype for the device is useful - but I would suggest that it is a cluster that can be reused in the protocol section of many observation archetypes. This will provide what you need without creating a participation for the agent - which you will need if the whole entry of the measurement is added as an automated procedure.

I understand that you want to change the way people look at data - but we already have a lot of literature to show that home measurements by non-professionals are better than those done by professionals (more rigorous about conditions, more careful, more relaxed and therefore more representative) so I don’t think that grabbing massive data collections will change the world the way you hope.

I would suggest that you contact Continua in the first instance and see what sort of feeds we can expect from the device industry - we probably should live with that for the moment.

Cheers, Sam

Stef Verlinden wrote:

8

Dear Stef,

Your concern about data quality is an important and difficult challenge.

With stand-alone provider systems, including paper records, there has been a tacit sense of trust of information acquired locally, and regrettably sometimes a mistrust of information acquired from outside unless from a known party. Well know examples of this, speaking now from personal memories of general practice, is the need of a consultant to repeat a blood test that I had organised and copied him even though it was performed very recently at his own hospital. Clinical trust is, on paper and electronically, to some extent idiosyncratic, variable and opinionated rather than formal, explicit and transparent.

With shared EHRs, we will be able to view data from multiple systems and settings in an increasingly seamless way, but we still need to be able to know if we can “trust” a data item in order to use it to inform care decisions.

Such trust, as you outline in your blood clotting (INR test) example, or for example the assertion of a new diagnosis, is multi-factorial.

Do I trust that the professional was qualified and trained to make the observation or assertion?
Do I trust that the facilities used by that professional were suitable (fit for purpose) and well maintained/calibrated?
Is there a clear rationale for why the observation/diagnosis was made (on what basis the diagnosis was made, or why was the test performed)?
Is that lab’s normal range the same as mine for this test?
If there is an explicit level of measurement accuracy, is this transparently declared?
If there is a formal expression of uncertainty, is this present and accessible to the reader? How might this be represented? Will the author and I share a common view of what “probable” means?
If there is any supplementary clinical commentary, such as cautions, uncertainties, possible confounding factors etc. are these documented and accessible?
If there was a protocol or guideline underpinning the decision, is this explicitly documented and referenced?
Is this data item in the correct patient’s record?
Is it correctly referencing the patient as the information subject, and is not actually a family history item?
Has this data item subsequently been revised/replaced because it was recorded in error?
etc.

Some of this information can be (and often is) explicitly documented in the EHR: e.g. the role and status of the author, references to a guideline, normal ranges, links between a new diagnosis and background clinical observations or test results, a clinical commentary if this was entered, version history.

Some elements are potentially available, but are not habitually included in the individual patient’s record today: e.g. laboratory calibration data, dates when devices were last checked. There is nothing in the EHR architecture to prevent this data being recorded, but these facts are not always known to the author. (Do we usually know when the BP sphygmomanometer we normally use was last checked, and if was it done by a suitably qualified person? - yes, if it has a sticker on it! Do we write this in each of our case notes alongside the BP?- never!)

Some of the trust information would be hard for an EHR system to know, and most authors will not wish to declare when entering data: recent education and training, how proficient they feel using an instrument, state formally how confident they feel about a diagnosis. Your proposal for levels of confidence would be nice if it could work, but all of the literature to date suggests that there is insufficient clinical consensus on the formalisation and terming of levels of confidence or certainty to permit a codeset to be used with any precision and inter-rater consistency. Even if such a code set were to emerge, there is quite a challenge in encouraging this kind of transparency. If it were adopted, however, this could easily be added to an EHR model.

We are presenty tacking some of this, not in relation to trust but in relation to clinical governance for anticoagulation services. We have agreed in north London that evidence of professional training, equipment and calibration, ability to use the clinical application and its decision support etc. will be reported on paper by each care site and not via our clinical systems. (Our EHR systems will only be asked to provide patient centric information for the governance reports.)

Do you feel that the clinical community should try to formalise an approach for explicit and consistently-usable expressions of clinical certainty and/or clinical proficiency?

With best wishes,

Dipak

9

Dear Dipak, Stef and all,

When sharing EHR data becomes to true, the accuracy of EHR data and people’s trust on EHR data will draw more attention than ever.

In current situation, clinical staff normally has their own adjustment on the reliability of external data (external data means the data that are not directly obtained by the clinical staff). They may/may not ask some questions before they use the external data. It is completely depend on the way they work and it is their responsibility to do the adjustment to decide whether to use the data or discard them. Additionally, the information about the data quality (e.g. clinical devices) is not normally recorded in the data. What I am trying to say is that the current clinical system is not quite ready for EHR sharing.

I certainly believe clinical community should have a standard way to model the data quality information for clinical data, such as who collected the data, role, device etc. Every local health service may have their own algorithms to inference the accuracy or there may be standard algorithms.

Technically, I think archetypes would be used to represent the details about clinical devices because the device information would be shared. The clinical archetypes can only capture the device id in the data quality cluster (we don’t want the same device information appears in each clinical archetype instance, do we?).

Cheers,
Chunlan

Dipak Kalra wrote:

10

Hi folks,
Here are my thoughts.
The accuracy of an observation depends on many variables that may or may not be quantifiable. It could be effected by the man, the machine, the methodology used, personal bias or even deliberate attempts at falsification.
So end of the day what do we, as physicians, do? I can tell you what I do. I base my trust on experience - my experience with the results that I get, over the months or years, from a Lab, specialist of paramedic. and although nothing is absolute, I find that results from certified Labs do turn out to be more consistant than those done by smaller wayside labs. What I'm stating applies to more than 60% of the population that lives in countries where certification of Labs and regular monitoring is not the norm.

Therefore we could add a field for the Lab's certification.

End of the day, most doctors do not go on Lab results alone; they depend on their own clinical assessment too, like I do.

Warm regards,

Dr D Lavanian
MBBS, MD
Vice President - Software Division
AxSys Healthtech Ltd
Mobile: +91-9949902800

11

Stef Verlinden wrote:

I don't want to be offensive but it doesn't. This is a probably
correct but for me way to technical answer to a practical question.

As a clinician I need to be able to establish that de a certain value
that is send to me is reliable.

so we need to know what 'reliable' means? See Dipak's reponse - it may
be a question of personal trust, organisational trust, of confidence in
equipment, of the information being current (not correct but out of
date) or many other things.

Therefore I need not only to know who measured this data but also
which device was used and was this device properly calibrated and
maintained at the time of measurement, was the performer properly
trained to do the measurement en was the meausurement protocol
followed. I know blood pressure isn't the best example so lets take a
more critical one like blood clotting time.

yes, so for these things.....

- who measured it is recorded in the Observation.provider (and maybe
participations) and in the Composition.context
- their identity and function in the measurement process are recorded in
the Participation object (Observation.provider);
Composition.context.participations
- which device & calibration info is recorded in the protocol part of
the Observation

whether the person who did the observation is professionally capable of
doing it is another matter not really controllable by the ICT alone
(other than by access control).

All this I want to check against our 'internal' protocol so that I
know whether I can use this data or discard it.
This is one of the major reasons (at least in the Netherlands) that
medical specialist discard data that already gathered by a GP
(everybody has to go to a GP first in the Netherlands. The GP examines
a patient and might perform some additional tests/ measurements before
he/she decides to refer to a specialist).

I think this points to a cultural phenomenon that needs to be addressed
outside the software. The most you can do is record who did it and what
their role was, which you can already do. You could certainly archetype
your confidence in their observation as a confidence attribute but then
it is a kind of clinical judgement on your part, and you may end up
having to create an Evaluation per Observation just to record this.

The other major reason is that data can't be exchanged between the
different systems.. One of the reasons that a EHR is promoted, is that
data can be shared and re-used. Especially the latter is of great
importance since many decisions makers firmly believe that's the way
to controls the expanding costs of the system. If we can share data
thanks to openEHR but still the data quality remains questionable (and
yes this has all to do with trust, but also with protecting ones
income/ position) people will use that argument to reject the data and
remain working the old fashioned way.

What I'm looking for is a practical way to establish this situation.
Any suggestions?

I doubt if modelling data reliability at the Observation level is the
right approach if the real problem is trust with certain providers /
people in your care network....if you are going to record that kind of
thing, I would have thought you needed some kind of trust / confidence
markers in your provider registry...

- thomas beale

12

Modeling TRUST is nonsensical thing.
Modeling DATA RELIABILITY is a nonsensical thing.

Both are complex concepts that partly depend on very subjective interpretations of people, processes, applications and hardware.
But they depend as well on objective data about people, processes, applications and hardware.

It is in this category that we must find ways to document the relevant objective co-determinants about people, processes, applications and hardware.
Archetypes are the structures that capture these objective co-determinants.

Stef’s question is: What can we capture in Archetypes when we document outcomes of machine (medical device) generated data, and the people that operated these machines.

Greetings

Gerard

– –
Gerard Freriks, MD
Huigsloterdijk 378
2158 LR Buitenkaag
The Netherlands

T: +31 252544896
M: +31 620347088
E: gfrer@luna.nl

Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

conexis

13

So that does that mean that archetypes should not contain any of the numerous scales that measure subjective data i.e. pain rating scales. I can’t see why any such limitation should be accepted.

Hugh Grady

14

Hi,

What Gerard expressed is that you can include any measure, but that
"reliability information" about this measurement is complex and may lead
to an infinite loop.

Complex because it typically includes 4 parties: the subject of care,
the information collector (a health professional or not), the device and
(usually later on) the one that uses this information.
The quality of the whole process can be more or less subject dependent,
device user dependent, device dependent and even information user dependent.

This "more or lessness" leads to the infinite loop : a 5th party will
decide that this process is "more or less reliable"... but doing this is
another process... and is also user dependent, etc...

Better stop before you need some Aspirin.

Regards,

Philippe Ameline

Hugh Grady wrote:

15

Hugh,

Do not misinterpret me.
We do need scales to express subjective things like pain.

But as far as medial devices (physical things) are concerned we only can record the hard physical facts, and wether the person operating it has been trained, etc.

Gerard

– –
Gerard Freriks, MD
Huigsloterdijk 378
2158 LR Buitenkaag
The Netherlands

T: +31 252544896
M: +31 620347088
E: gfrer@luna.nl

Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

16

If I remember correct, the CONTSYS part 1 standard document "System of
concepts to support continuity of care" distinguish between
'non ratified clinical data' (clinical data, the relevance of which has
not been ratified by a health care professional)
and
'clinical data for import' (data that is candidate for import into an EHR
[..] after a health care professional has ratified its clinical
relevance).

Shouldn't it be possible to utilize the mechanism that these definitions
imply to solve the problem with importing representations of observations
and / or measurements recorded by the patient himself (e.g. in a personal
health record) or by other healthcare providers (that may or may not be
trustworthy)?

The problem of creating representations of patient's subjective
experiences is also an ontological problem. Are subjective experiences
real phenomena (e.g chest pain in acute myocardial infarction) , or is it
their reference (the exact location and magnitude of the myocardial
infarction) that should be recorded?

regards
Arild Faxvaag

Sitat Gerard Freriks <gfrer@luna.nl>:

17

Hugh Grady wrote:

So that does that mean that archetypes should not contain any of the
numerous scales that measure subjective data i.e. pain rating scales.
I can’t see why any such limitation should be accepted.

no - that is no problem. Recording such things is in the normal purview
of the clinician - they are just recording what the patient says. The
clinician can also record a level of confidence in the statements, if
that was designed into the archetype, or other statements that indicate
he doesn't think the patient is telling the truth....

The problem being discussed here, as far as I understand it is not the
level of confidence in the individual or machine from which the data
were taken, but from the organisation providing a whole stream of health
information, i.e. 'we inherently have a low level of confidence in
health data coming from acme health partners'. Or at least this is part
of the problem.

- thomas

19

Gerard,

With regard to your statement that ‘as far as medial devices (physical things) are concerned we only can record the hard physical facts, and wether the person operating it has been trained, etc.’. Why is this? Why can’t we record higher level (more abstract) measurements or beliefs on the reliability or trustworthiness of particular machines, even if these measurements include a subjective element. Clearly such measurements are not foolproof (is any data?), but is it not better than nothing? It would still be up to the user of the data to decide if it is of sufficient value to be useful for their purposes. If they require data backed up by precise measurements of only ‘objective co-determinants’ then so be it, but aren’t there situations where that is not so?

regards,

Hugh

20

I suggest it would be better to concentrate first on modelling the real world - implement features that clinicians already record.

Does anyone record beliefs now?

It would be appropriate to record a suspicion that a device is not trustworthy, with the communication to whoever maintains it, with the response later and the retrospective amendment of measurement value or quality. Is the eHR ready to handle that?

regards,
Colin