# Data quality questions/ proposal **Category:** [Clinical (archive)](https://discourse.openehr.org/c/clinical-archive/153) **Created:** 2007-07-10 09:41 UTC **Views:** 7 **Replies:** 45 **URL:** https://discourse.openehr.org/t/data-quality-questions-proposal/14665 --- ## Post #1 by @Stef_Verlinden1 One of the major requirements we have is what I call a ‘data quality marker’. So the blood pressure recorded is 88/124 but what is the ‘value/ quality’ of this measurement. IMHO any recorded value is useless unless the quality of this measurement can be established and taken into account when interpreting the data In order to establish this data quality we need to add some attributes to the observation archetypes used to record such measurement. So far as we can see now we think that these attributes are a data quality field and a device/instrument reference (which requires a device archetype) and this is what we would like to propose to the community. Since I don’t know exactly how to do that and we still have many unanswered questions I’ll describe what we’re thinking about. It’s very well possible that these thing are already in place, in that case we’re aren’t aware of that and would like to be pointed in the right direction. In our ‘model’ data quality can be described as: excellent, good, doubtful and insufficient. Here the first hurdle arises: one needs a protocol to define what is excellent, good etc. These are probably ‘local’ criteria, so the can’t be embedded in a general archetype. Our idea is to create a specialisation of the observation archetype in question, in which the local protocol is attached. For instance this blood pressure archetype with the local Dutch data quality criteria would be openEHR-EHR-OBSERVATION.blood_pressure-data_qualityNL.v1.adl To give an example these are the criteria for blood pressure we’re thinking off: Excellent: data measured/obtained by a qualified healthcare provider, with a certified instrument/device that’s calibrated against a ‘golden standard’, the measurement error is within a tight bandwidth (<5%), the validity duration of the calibration isn’t expired, maintained on time and by qualified personal (This can’t be met when self-measuring in the home situation) Good: data measured/obtained by a qualified person (this can also be a properly trained patient/citizen), with a certified (CE marked) instrument/device that’s self calibrating, the measurement error is within a tight bandwidth (I.e. machine is approved by the European society of Hypertension (ESH), the machine isn’t broke and functioning well ## Poor/ Doubtfuldata measured/obtained by a qualified person (this can also be a properly trained patient/citizen), with a certified instrument/device that’s self calibrating, the measurement error isn’t within a tight bandwidth (CE marking alone allows measurement errors >7%), the machine isn’t broke and functioning well Insufficient: in all the other situations As a consequence we need to add at least one other attribute: a reference/ link to the device used. In our opinion there should be a separate archetype for a device/instrument. In this archetypes not only the unique identifiers of this device are recorded but also information about calibration, maintenance etc. etc. So far as I understand/can see such an archetype doesn’t exist today. Our idea is to use the demographic archetype model for this. In fact there is already a demographic archetype subtype for ‘agents’. So either we extent this subclass so it can be used for devices or we create a new archetype class for devices/instruments based on this agent model. Another thing that is already established is the capability of a healthcare professional. I.e. is this person properly trained to operate a device/instrument? In that respect I would like to add similar capabilities for non-health care professionals. In the above case patients/ citizens also can measure their own blood pressure. Before they can do that, they’re trained and examined. Only then they’re capable of producing ‘good quality’ (provided that they meet the other criteria as well) data. Can anybody please comment on this? As stated before it would be really of great help if we could organise some sort of ‘archetype boot camp’ to create an expanding community of clinicians who know how to create archetypes and harmonize the ‘wishes and ideas’ that will come up as soon as more people start creating and using archetypes. Cheers, Stef --- ## Post #2 by @ian.mcnicoll Hi Stef, Very interesting and, in principle, quite correct but I am not sure how practicable this would be in the real world. There are so many variables that might determine whether a Blood pressure is either accurate or appropriate, including the purpose for which the historical blood pressure reading is being monitored or reviewed. For instance a series of post-op Blood pressures may be accurately taken but be quite inappropriate to use for long term hypertension monitoring. The other problem is that you can only make some of these measures of data quality in retrospect e.g. You may have a BP device that has been calibrated correctly but later malfunctions. Or you may have a seemingly competent patient who turns out to have been messing up the process. I can see some value in a simple flag (defaulting to false) to identify BP readings that should not be used for monitoring purposes because they are known to have quality issues or were taken in inappropriate circumstances e.g post-op or severe illness but I think your data quality markers may be too complex to be workable. Regards, Ian McNicoll Medical Informatics --- ## Post #3 by @thomas.beale We had many discussions over the years on this topic in openEHR, and eventually decided to get rid of built in flags to do with certainty. There are ways to express certainty (see below) but we need to understand the problem first. This is our current understanding. - there is a concept of 'accuracy' which is the level of error in instruments, other objective measuring mechanisms. This is modelled in the Quantity data types, see the Data Types IM ([http://svn.openehr.org/specification/TAGS/Release-1.0.1/publishing/architecture/rm/data_types_im.pdf](http://svn.openehr.org/specification/TAGS/Release-1.0.1/publishing/architecture/rm/data_types_im.pdf)) for a detailed discussion. Mathematically accuracy defines a confidence interval into which the value on a dial, readout is in; mechanically it might correspond to some tolerances in manufacture etc. The main thing is that 'accuracy' is a concept of objective error. This concept seems to apply pretty much to quantitative values, including ordinals, and is this accommodated in the Quantity package. There is also a magnitude_status attribute which allows ~, <, >, <=, >= to be used, since some lab software generates such markers. - there is a concept of 'confidence' in a clinical judgement, e.g. being 75% sure about a diagnosis (which is really a differential diagnosis), risk of a problem occurring in a patient etc. This information can be archetyped in an Evaluation archetype, and there is no standard for it - some people use percentages, some use confidence - very low/low/medium/high/very high, there are other schemes, and there have been studies on how terms like high/medium/low correspond to percentages - which show that doctors don't all equate them to the same numbers. - either of these can be adjusted in restrospect, using versioning (built in to openEHR semantics). After some 5+ years of debating this issue in a very wide context, including on these lists, I think the current approach is probably about right, but some may not be surprised to know that some of us were just as certain years ago of the need for a 'flag' as you are now...but the more you try using it, the less it works. So we have restricted such a flag + accuracy to Quantity types, and otherwise confidence is archetyped. hope this helps. - thomas beale Ian McNicoll wrote: --- ## Post #4 by @Stef_Verlinden1 > Hi Stef, > > Very interesting and, in principle, quite correct but I am not sure how practicable this would be in the real world. There are so many variables that might determine whether a Blood pressure is either accurate or appropriate, including the purpose for which the historical blood pressure reading is being monitored or reviewed. For instance a series of post-op Blood pressures may be accurately taken but be quite inappropriate to use for long term hypertension monitoring. I totally agree. That's why I think that such protocols should be (local) specialisations of a otherwise generic basic archetype. Altough bloodpressure isn't a 'critical' value, in my opinion it's critical to be able to establish data quality when using data of 'others'. At least here in the Netherlands, one inherits the responsibility for the use of it, when one chooses to deploy data of poor quality, f.i. in the treatment of a patient. If the patient dies because of this both the provider and the user of the data are held equally responsible. If data can't be used because of the lack of quality, what's the use for an EHR? > The other problem is that you can only make some of these measures of data quality in retrospect e.g. You may have a BP device that has been calibrated correctly but later malfunctions. Or you may have a seemingly competent patient who turns out to have been messing up the process. That's why we need a seperate device/instrument archetype which records things as calibration en maintainance of the device. From that archetype the 'actual' status of the device can be dertermined. We envision apllacations in which a reminder/ warning will show up when a device doesn't meet the criteria and prevents data of being entered in the system. The beauty of archetypes is that one also can 'go back in time'. So f.i. in case of a dispute i t's possible to 'establish' the device status at any moment. > I can see some value in a simple flag (defaulting to false) to identify BP readings that should not be used for monitoring purposes because they are known to have quality issues or were taken in inappropriate circumstances e.g post-op or severe illness but I think your data quality markers may be too complex to be workable. Coming from a semi- pharmaceutical back ground it really surprises me how high quality standards there are for the development of any drug and how these standards largely disappear as soon as health care providers get involved. It looks like the fact that something is done by a doctor is sufficient to prove the quality of it:-) Cheers, Stef --- ## Post #5 by @Seref Dear all, I hope I am not taking your time with an irrelevant subject, but US health and Human services has posted a report on the prototype architectures, and this seemed like a good place to share this info\. My apologies if this is off\-topic or a duplicate post\. http://www.hhs.gov/healthit/healthnetwork/resources/summary_report_on_nhin_Prototype_architectures.pdf Best Regards Seref Arikan --- ## Post #6 by @Stef_Verlinden1 > We had many discussions over the years on this topic in openEHR, and eventually decided to get rid of built in flags to do with certainty. There are ways to express certainty (see below) but we need to understand the problem first. This is our current understanding. > > - there is a concept of 'accuracy' which is the level of error in instruments, other objective measuring mechanisms. This is modelled in the Quantity data types, see the Data Types IM ([http://svn.openehr.org/specification/TAGS/Release-1.0.1/publishing/architecture/rm/data_types_im.pdf](http://svn.openehr.org/specification/TAGS/Release-1.0.1/publishing/architecture/rm/data_types_im.pdf)) for a detailed discussion. Mathematically accuracy defines a confidence interval into which the value on a dial, readout is in; mechanically it might correspond to some tolerances in manufacture etc. The main thing is that 'accuracy' is a concept of objective error. This concept seems to apply pretty much to quantitative values, including ordinals, and is this accommodated in the Quantity package. There is also a magnitude_status attribute which allows ~, <, >, <=, >= to be used, since some lab software generates such markers. > > - there is a concept of 'confidence' in a clinical judgement, e.g. being 75% sure about a diagnosis (which is really a differential diagnosis), risk of a problem occurring in a patient etc. This information can be archetyped in an Evaluation archetype, and there is no standard for it - some people use percentages, some use confidence - very low/low/medium/high/very high, there are other schemes, and there have been studies on how terms like high/medium/low correspond to percentages - which show that doctors don't all equate them to the same numbers. > - either of these can be adjusted in restrospect, using versioning (built in to openEHR semantics). > After some 5+ years of debating this issue in a very wide context, including on these lists, I think the current approach is probably about right, but some may not be surprised to know that some of us were just as certain years ago of the need for a 'flag' as you are now...but the more you try using it, the less it works. So we have restricted such a flag + accuracy to Quantity types, and otherwise confidence is archetyped. > > hope this helps. I don't want to be offensive but it doesn't. This is a probably correct but for me way to technical answer to a practical question. As a clinician I need to be able to establish that de a certain value that is send to me is reliable. Therefore I need not only to know who measured this data but also which device was used and was this device properly calibrated and maintained at the time of measurement, was the performer properly trained to do the measurement en was the meausurement protocol followed. I know blood pressure isn't the best example so lets take a more critical one like blood clotting time. All this I want to check against our 'internal' protocol so that I know whether I can use this data or discard it. This is one of the major reasons (at least in the Netherlands) that medical specialist discard data that already gathered by a GP (everybody has to go to a GP first in the Netherlands. The GP examines a patient and might perform some additional tests/ measurements before he/she decides to refer to a specialist). The other major reason is that data can't be exchanged between the different systems.. One of the reasons that a EHR is promoted, is that data can be shared and re-used. Especially the latter is of great importance since many decisions makers firmly believe that's the way to controls the expanding costs of the system. If we can share data thanks to openEHR but still the data quality remains questionable (and yes this has all to do with trust, but also with protecting ones income/ position) people will use that argument to reject the data and remain working the old fashioned way. What I'm looking for is a practical way to establish this situation. Any suggestions? Cheers, Stef --- ## Post #7 by @Sam Hi Stef I think we can do some things that will be helpful. **Accuracy** - As Tom has said we can add some accuracy (which can be a %) to any reading, so it is possible for devices to give a reading with an accuracy. This appears to cover one need that you have - we need to see what is happening with the device standards on this front from the Continua group and see how they propose to deal with this. [http://www.continuaalliance.org](http://www.continuaalliance.org) **Device and callibration** - You and others can develop a terminology for home measuring devices - perhaps the Continua group will provide this on asking. This can be linked to a data store of the measuring characteristics of the device. This can be set as a default value within each users template - unlikely to be entered manually every time. - You can ask for the date of last callibration of the device (perhaps who did it) - although this is unlikely to be filled in unless it comes as an automatic feed from the device. Your idea for an archetype for the device is useful - but I would suggest that it is a cluster that can be reused in the protocol section of many observation archetypes. This will provide what you need without creating a participation for the agent - which you will need if the whole entry of the measurement is added as an automated procedure. I understand that you want to change the way people look at data - but we already have a lot of literature to show that home measurements by non-professionals are better than those done by professionals (more rigorous about conditions, more careful, more relaxed and therefore more representative) so I don't think that grabbing massive data collections will change the world the way you hope. I would suggest that you contact Continua in the first instance and see what sort of feeds we can expect from the device industry - we probably should live with that for the moment. Cheers, Sam Stef Verlinden wrote: --- ## Post #8 by @Dipak_Kalra Dear Stef, Your concern about data quality is an important and difficult challenge. With stand-alone provider systems, including paper records, there has been a tacit sense of trust of information acquired locally, and regrettably sometimes a mistrust of information acquired from outside unless from a known party. Well know examples of this, speaking now from personal memories of general practice, is the need of a consultant to repeat a blood test that I had organised and copied him even though it was performed very recently at his own hospital. Clinical trust is, on paper and electronically, to some extent idiosyncratic, variable and opinionated rather than formal, explicit and transparent. With shared EHRs, we will be able to view data from multiple systems and settings in an increasingly seamless way, but we still need to be able to know if we can "trust" a data item in order to use it to inform care decisions. Such trust, as you outline in your blood clotting (INR test) example, or for example the assertion of a new diagnosis, is multi-factorial. Do I trust that the professional was qualified and trained to make the observation or assertion? Do I trust that the facilities used by that professional were suitable (fit for purpose) and well maintained/calibrated? Is there a clear rationale for why the observation/diagnosis was made (on what basis the diagnosis was made, or why was the test performed)? Is that lab's normal range the same as mine for this test? If there is an explicit level of measurement accuracy, is this transparently declared? If there is a formal expression of uncertainty, is this present and accessible to the reader? How might this be represented? Will the author and I share a common view of what "probable" means? If there is any supplementary clinical commentary, such as cautions, uncertainties, possible confounding factors etc. are these documented and accessible? If there was a protocol or guideline underpinning the decision, is this explicitly documented and referenced? Is this data item in the correct patient's record? Is it correctly referencing the patient as the information subject, and is not actually a family history item? Has this data item subsequently been revised/replaced because it was recorded in error? etc. Some of this information can be (and often is) explicitly documented in the EHR: e.g. the role and status of the author, references to a guideline, normal ranges, links between a new diagnosis and background clinical observations or test results, a clinical commentary if this was entered, version history. Some elements are potentially available, but are not habitually included in the individual patient's record today: e.g. laboratory calibration data, dates when devices were last checked. There is nothing in the EHR architecture to prevent this data being recorded, but these facts are not always known to the author. (Do we usually know when the BP sphygmomanometer we normally use was last checked, and if was it done by a suitably qualified person? - yes, if it has a sticker on it! Do we write this in each of our case notes alongside the BP?- never!) Some of the trust information would be hard for an EHR system to know, and most authors will not wish to declare when entering data: recent education and training, how proficient they feel using an instrument, state formally how confident they feel about a diagnosis. Your proposal for levels of confidence would be nice if it could work, but all of the literature to date suggests that there is insufficient clinical consensus on the formalisation and terming of levels of confidence or certainty to permit a codeset to be used with any precision and inter-rater consistency. Even if such a code set were to emerge, there is quite a challenge in encouraging this kind of transparency. If it were adopted, however, this could easily be added to an EHR model. We are presenty tacking some of this, not in relation to trust but in relation to clinical governance for anticoagulation services. We have agreed in north London that evidence of professional training, equipment and calibration, ability to use the clinical application and its decision support etc. will be reported on paper by each care site and not via our clinical systems. (Our EHR systems will only be asked to provide patient centric information for the governance reports.) Do you feel that the clinical community should try to formalise an approach for explicit and consistently-usable expressions of clinical certainty and/or clinical proficiency? With best wishes, Dipak --- ## Post #9 by @chunlan.ma Dear Dipak, Stef and all, When sharing EHR data becomes to true, the accuracy of EHR data and people's trust on EHR data will draw more attention than ever. In current situation, clinical staff normally has their own adjustment on the reliability of external data (external data means the data that are not directly obtained by the clinical staff). They may/may not ask some questions before they use the external data. It is completely depend on the way they work and it is their responsibility to do the adjustment to decide whether to use the data or discard them. Additionally, the information about the data quality (e.g. clinical devices) is not normally recorded in the data. What I am trying to say is that the current clinical system is not quite ready for EHR sharing. I certainly believe clinical community should have a standard way to model the data quality information for clinical data, such as who collected the data, role, device etc. Every local health service may have their own algorithms to inference the accuracy or there may be standard algorithms. Technically, I think archetypes would be used to represent the details about clinical devices because the device information would be shared. The clinical archetypes can only capture the device id in the data quality cluster (we don't want the same device information appears in each clinical archetype instance, do we?). Cheers, Chunlan Dipak Kalra wrote: --- ## Post #10 by @lavanian Hi folks, Here are my thoughts\. The accuracy of an observation depends on many variables that may or may not be quantifiable\. It could be effected by the man, the machine, the methodology used, personal bias or even deliberate attempts at falsification\. So end of the day what do we, as physicians, do? I can tell you what I do\. I base my trust on experience \- my experience with the results that I get, over the months or years, from a Lab, specialist of paramedic\. and although nothing is absolute, I find that results from certified Labs do turn out to be more consistant than those done by smaller wayside labs\. What I'm stating applies to more than 60% of the population that lives in countries where certification of Labs and regular monitoring is not the norm\. Therefore we could add a field for the Lab's certification\. End of the day, most doctors do not go on Lab results alone; they depend on their own clinical assessment too, like I do\. Warm regards, Dr D Lavanian MBBS, MD Vice President \- Software Division AxSys Healthtech Ltd Mobile: \+91\-9949902800 --- ## Post #11 by @thomas.beale Stef Verlinden wrote: > > I don't want to be offensive but it doesn't\. This is a probably > correct but for me way to technical answer to a practical question\. > > As a clinician I need to be able to establish that de a certain value > that is send to me is reliable\. so we need to know what 'reliable' means? See Dipak's reponse \- it may be a question of personal trust, organisational trust, of confidence in equipment, of the information being current \(not correct but out of date\) or many other things\. > Therefore I need not only to know who measured this data but also > which device was used and was this device properly calibrated and > maintained at the time of measurement, was the performer properly > trained to do the measurement en was the meausurement protocol > followed\. I know blood pressure isn't the best example so lets take a > more critical one like blood clotting time\. yes, so for these things\.\.\.\.\. \- who measured it is recorded in the Observation\.provider \(and maybe participations\) and in the Composition\.context \- their identity and function in the measurement process are recorded in the Participation object \(Observation\.provider\); Composition\.context\.participations \- which device & calibration info is recorded in the protocol part of the Observation whether the person who did the observation is professionally capable of doing it is another matter not really controllable by the ICT alone \(other than by access control\)\. > All this I want to check against our 'internal' protocol so that I > know whether I can use this data or discard it\. > This is one of the major reasons \(at least in the Netherlands\) that > medical specialist discard data that already gathered by a GP > \(everybody has to go to a GP first in the Netherlands\. The GP examines > a patient and might perform some additional tests/ measurements before > he/she decides to refer to a specialist\)\. I think this points to a cultural phenomenon that needs to be addressed outside the software\. The most you can do is record who did it and what their role was, which you can already do\. You could certainly archetype your confidence in their observation as a confidence attribute but then it is a kind of clinical judgement on your part, and you may end up having to create an Evaluation per Observation just to record this\. > The other major reason is that data can't be exchanged between the > different systems\.\. One of the reasons that a EHR is promoted, is that > data can be shared and re\-used\. Especially the latter is of great > importance since many decisions makers firmly believe that's the way > to controls the expanding costs of the system\. If we can share data > thanks to openEHR but still the data quality remains questionable \(and > yes this has all to do with trust, but also with protecting ones > income/ position\) people will use that argument to reject the data and > remain working the old fashioned way\. > > What I'm looking for is a practical way to establish this situation\. > Any suggestions? I doubt if modelling data reliability at the Observation level is the right approach if the real problem is trust with certain providers / people in your care network\.\.\.\.if you are going to record that kind of thing, I would have thought you needed some kind of trust / confidence markers in your provider registry\.\.\. \- thomas beale --- ## Post #12 by @system Modeling TRUST is nonsensical thing. Modeling DATA RELIABILITY is a nonsensical thing. Both are complex concepts that partly depend on very subjective interpretations of people, processes, applications and hardware. But they depend as well on objective data about people, processes, applications and hardware. It is in this category that we must find ways to document the relevant objective co-determinants about people, processes, applications and hardware. Archetypes are the structures that capture these objective co-determinants. Stef's question is: What can we capture in Archetypes when we document outcomes of machine (medical device) generated data, and the people that operated these machines. Greetings Gerard -- -- Gerard Freriks, MD Huigsloterdijk 378 2158 LR Buitenkaag The Netherlands T: +31 252544896 M: +31 620347088 E: [gfrer@luna.nl](mailto:gfrer@luna.nl) Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755 **conexis** --- ## Post #13 by @Hugh_Grady So that does that mean that archetypes should not contain any of the numerous scales that measure subjective data i.e. pain rating scales. I can’t see why any such limitation should be accepted. Hugh Grady --- ## Post #14 by @Philippe_AMELINE Hi, What Gerard expressed is that you can include any measure, but that "reliability information" about this measurement is complex and may lead to an infinite loop\. Complex because it typically includes 4 parties: the subject of care, the information collector \(a health professional or not\), the device and \(usually later on\) the one that uses this information\. The quality of the whole process can be more or less subject dependent, device user dependent, device dependent and even information user dependent\. This "more or lessness" leads to the infinite loop : a 5th party will decide that this process is "more or less reliable"\.\.\. but doing this is another process\.\.\. and is also user dependent, etc\.\.\. Better stop before you need some Aspirin\. Regards, Philippe Ameline Hugh Grady wrote: --- ## Post #15 by @system Hugh, Do not misinterpret me. We do need scales to express subjective things like pain. But as far as medial devices (physical things) are concerned we only can record the hard physical facts, and wether the person operating it has been trained, etc. Gerard -- -- Gerard Freriks, MD Huigsloterdijk 378 2158 LR Buitenkaag The Netherlands T: +31 252544896 M: +31 620347088 E: [gfrer@luna.nl](mailto:gfrer@luna.nl) Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755 --- ## Post #16 by @Arild_Faxvaag1 If I remember correct, the CONTSYS part 1 standard document "System of concepts to support continuity of care" distinguish between 'non ratified clinical data' \(clinical data, the relevance of which has not been ratified by a health care professional\) and 'clinical data for import' \(data that is candidate for import into an EHR \[\.\.\] after a health care professional has ratified its clinical relevance\)\. Shouldn't it be possible to utilize the mechanism that these definitions imply to solve the problem with importing representations of observations and / or measurements recorded by the patient himself \(e\.g\. in a personal health record\) or by other healthcare providers \(that may or may not be trustworthy\)? The problem of creating representations of patient's subjective experiences is also an ontological problem\. Are subjective experiences real phenomena \(e\.g chest pain in acute myocardial infarction\) , or is it their reference \(the exact location and magnitude of the myocardial infarction\) that should be recorded? regards Arild Faxvaag Sitat Gerard Freriks <gfrer@luna\.nl>: --- ## Post #17 by @thomas.beale Hugh Grady wrote: > > So that does that mean that archetypes should not contain any of the > numerous scales that measure subjective data i\.e\. pain rating scales\. > I can’t see why any such limitation should be accepted\. > no \- that is no problem\. Recording such things is in the normal purview of the clinician \- they are just recording what the patient says\. The clinician can also record a level of confidence in the statements, if that was designed into the archetype, or other statements that indicate he doesn't think the patient is telling the truth\.\.\.\. The problem being discussed here, as far as I understand it is not the level of confidence in the individual or machine from which the data were taken, but from the organisation providing a whole stream of health information, i\.e\. 'we inherently have a low level of confidence in health data coming from acme health partners'\. Or at least this is part of the problem\. \- thomas --- ## Post #18 by @Sam --- ## Post #19 by @Hugh_Grady Gerard, With regard to your statement that ‘as far as medial devices (physical things) are concerned we only can record the hard physical facts, and wether the person operating it has been trained, etc.’. Why is this? Why can’t we record higher level (more abstract) measurements or beliefs on the reliability or trustworthiness of particular machines, even if these measurements include a subjective element. Clearly such measurements are not foolproof (is any data?), but is it not better than nothing? It would still be up to the user of the data to decide if it is of sufficient value to be useful for their purposes. If they require data backed up by precise measurements of only ‘objective co-determinants’ then so be it, but aren’t there situations where that is not so? regards, Hugh --- ## Post #20 by @Colin_Sutton I suggest it would be better to concentrate first on modelling the real world - implement features that clinicians already record. Does anyone record beliefs now? It would be appropriate to record a suspicion that a device is not trustworthy, with the communication to whoever maintains it, with the response later and the retrospective amendment of measurement value or quality. Is the eHR ready to handle that? regards, Colin --- ## Post #21 by @thomas.beale Colin Sutton wrote: > I suggest it would be better to concentrate first on modelling the > real world \- implement features that clinicians already record\. to be sure, this has been the focus of openEHR to date, with a lot of input from various specialties including pathology lab to ensure we have good coverage\. > > Does anyone record beliefs now? > > It would be appropriate to record a suspicion that a device is not > trustworthy, with the communication to whoever maintains it, with the > response later and the retrospective amendment of measurement value or > quality\. Is the eHR ready to handle that? yes, you can commit a new version of a result, and include as the reason 'error due to defective device' or whatever it may be\. \- thomas beale --- ## Post #22 by @ian.mcnicoll Thomas, As well as a textual reason, there may be a more generic requirement to flag 'not thought suitable for decision support or secondary uses', so that the result \(or entry\) is ignored by machine processing\. The standard practice in UK GP systems, where a Read coded item proves to be unreliable, is to 'delete' the original and recreate it as a Text only uncoded item\. This is a fascinating discussion\!\! It raises all sorts of issues in my head\. In particular, Stef he suggests that without reliable quality/trust markers, there will be limitations to the vision of the single homogenous EHR\. Whilst In principle, I think he is correct, I think such limitations are inevitable and as with others who have commented, feel that the cost and effort of maintaining a \*meaningful\* quality marker system would far outweigh the benefit\. Ian --- ## Post #23 by @thomas.beale Hi Arild, Arild Faxvaag wrote: > > Shouldn't it be possible to utilize the mechanism that these definitions > imply to solve the problem with importing representations of observations > and / or measurements recorded by the patient himself \(e\.g\. in a personal > health record\) or by other healthcare providers \(that may or may not be > trustworthy\)? >   as I have said elsewhere, I think the right approach is that trust or other indicators, if they relate to providers or organisations, belong in the demographics / provider registry\. > The problem of creating representations of patient's subjective > experiences is also an ontological problem\. Are subjective experiences > real phenomena \(e\.g chest pain in acute myocardial infarction\) , or is it > their reference \(the exact location and magnitude of the myocardial > infarction\) that should be recorded? >   our point of view is that the clinician records what is observed or said\. It is up to the specialist to make the right kind of investigations, including asking the right questions\. For example, as a doctor you know that you sometimes get a handle on pain if it is in a joint by flexing/extending in certain ways \- there is some subjectivity in how loud the patient yells when you twist their wrist, but by and large you know what is going on\. Headache and dull pains in the thorax or abdomen I presume are somewhat harder to characterise, but there are ways: does the headache prevent you from reading, working, sleeping, etc\. The MDs will know the appropriate protocols for all this\. In the end, the EHR can only act as a memory device for competent professionals \(and competent patients\) to record what they sense from the object of study \(the patient\)\. It can't know any absolute truths\. \- thomas beale --- ## Post #24 by @thomas.beale Ian McNicoll wrote: > Thomas, > > As well as a textual reason, there may be a more generic requirement to flag > 'not thought suitable for decision support or secondary uses', so that the > result \(or entry\) is ignored by machine processing\. The standard practice in > UK GP systems, where a Read coded item proves to be unreliable, is to > 'delete' the original and recreate it as a Text only uncoded item\. > yes, that has long been the thinking among people doing openEHR, CEN and related things\. > This is a fascinating discussion\!\! > > It raises all sorts of issues in my head\. In particular, Stef he suggests > that without reliable quality/trust markers, there will be limitations to > the vision of the single homogenous EHR\. Whilst > In principle, I think he is correct, I think such limitations are inevitable > and as with others who have commented, feel that the cost and effort of > maintaining a \*meaningful\* quality marker system would far outweigh the > benefit\. >   if we look at it in a cost\-benefit mode, then at what point do the costs of ignoring data from another source outweigh the supposed benefits? Ultimately quality of care and patient safety will become a question; if not then it is just academic anyway\. \- thomas --- ## Post #25 by @Sweetman_Pauline_NHS Ian, Allow me to join in on this one\. The issues that it raises in my head are 1\. provenance \(who said it, when and why\)\. This gives the clinican clues about how 'trustworthy' the information is\. 2\. credibility \(did they understand the implications of what they were recording, were they being mischievous, could they have misunderstood or misheard\) 3\. relevance to the patient care decisions that may be made in the future \(and therefore how necessary is it \(or not\) to be an input into the decision support engine\) NB\. 3 is not a static thing, relevance may change at some future time with a change in the patient's condition\. What was thought not to be relevant may become so e\.g\. hallucinations as a side effect may turn out to be \(or to cause\) a psychotic event instead\. Other discussions \(elsewhere\) regarding the relevance of a reaction to a drug and severity of a reaction are related to these issues inasmuch as one severe reaction of one type may be much less relevant for future care than a mild reaction of another type\. E\.g\. severe constipation may be much less important than mild bronchospasm\. As you say, an interesting discussion Pauline Sweetman --- ## Post #26 by @William_E_Hammond Stef, I would like to share your e\-mail with ACMi to see what responses we get\. It is an interesting proposition\. Question, why not just require excellent/good measurement? Ed Hammond              Stef Verlinden              <stef@vivici\.nl>              Sent by: To              openehr\-clinical\- For openEHR clinical discussions              bounces@openehr\.o <openehr\-clinical@openehr\.org>              rg cc                                                                                                                                                Subject              07/10/2007 05:41 Data quality questions/ proposal              AM                                                                                          Please respond to                 For openEHR                  clinical                 discussions              <openehr\-clinical                @openehr\.org>                                                                             One of the major requirements we have is what I call a ‘data quality marker’\. So the blood pressure recorded is 88/124 but what is the ‘value/ quality’ of this measurement\. IMHO any recorded value is useless unless the quality of this measurement can be established and taken into account when interpreting the data In order to establish this data quality we need to add some attributes to the observation archetypes used to record such measurement\. So far as we can see now we think that these attributes are a data quality field and a device/instrument reference \(which requires a device archetype\) and this is what we would like to propose to the community\. Since I don’t know exactly how to do that and we still have many unanswered questions I’ll describe what we’re thinking about\. It’s very well possible that these thing are already in place, in that case we’re aren’t aware of that and would like to be pointed in the right direction\. In our ‘model’ data quality can be described as: excellent, good, doubtful and insufficient\. Here the first hurdle arises: one needs a protocol to define what is excellent, good etc\. These are probably ‘local’ criteria, so the can’t be embedded in a general archetype\. Our idea is to create a specialisation of the observation archetype in question, in which the local protocol is attached\. For instance this blood pressure archetype with the local Dutch data quality criteria would be openEHR\-EHR\-OBSERVATION\.blood\_pressure\-data\_qualityNL\.v1\.adl To give an example these are the criteria for blood pressure we’re thinking off: Excellent: data measured/obtained by a qualified healthcare provider, with a certified instrument/device that’s calibrated against a ‘golden standard’, the measurement error is within a tight bandwidth \(<5%\), the validity duration of the calibration isn’t expired, maintained on time and by qualified personal \(This can’t be met when self\-measuring in the home situation\) Good: data measured/obtained by a qualified person \(this can also be a properly trained patient/citizen\), with a certified \(CE marked\) instrument/device that’s self calibrating, the measurement error is within a tight bandwidth \(I\.e\. machine is approved by the European society of Hypertension \(ESH\), the machine isn’t broke and functioning well Poor/ Doubtful data measured/obtained by a qualified person \(this can also be a properly trained patient/citizen\), with a certified instrument/device that’s self calibrating, the measurement error isn’t within a tight bandwidth \(CE marking alone allows measurement errors >7%\), the machine isn’t broke and functioning well Insufficient: in all the other situations As a consequence we need to add at least one other attribute: a reference/ link to the device used\. In our opinion there should be a separate archetype for a device/instrument\. In this archetypes not only the unique identifiers of this device are recorded but also information about calibration, maintenance etc\. etc\. So far as I understand/can see such an archetype doesn’t exist today\. Our idea is to use the demographic archetype model for this\. In fact there is already a demographic archetype subtype for ‘agents’\. So either we extent this subclass so it can be used for devices or we create a new archetype class for devices/instruments based on this agent model\. Another thing that is already established is the capability of a healthcare professional\. I\.e\. is this person properly trained to operate a device/instrument? In that respect I would like to add similar capabilities for non\-health care professionals\. In the above case patients/ citizens also can measure their own blood pressure\. Before they can do that, they’re trained and examined\. Only then they’re capable of producing ‘good quality’ \(provided that they meet the other criteria as well\) data\. Can anybody please comment on this? As stated before it would be really of great help if we could organise some sort of ‘archetype boot camp’ to create an expanding community of clinicians who know how to create archetypes and harmonize the ‘wishes and ideas’ that will come up as soon as more people start creating and using archetypes\. Cheers, Stef --- ## Post #27 by @William_E_Hammond I agree with these thoughts\. Several years ago, I developed an EHR for a SICU\. Altho patients were automatically monitored and blood pressure measured periodically, nurses also entered a manual blood pressure\. I asked why \- the reason the BP machines were inaccurate\. The algorithm I used to record blood pressure from the machines was to look at actualy value as well as trend data\. I was relatively easy to recognize BPs that ere artifiacts\. I monitored readings every 5 minutes and took an average for the hourly reading\. When compared with the nurses, the readings were always within 3 mm \- except for one machine that had an error and was replaced\. Lesson \- interesting question\. In my work now, I am proposing that we receive data fromother sites and duplicate the test as we now do\. Then we compare the results\. If they are equivalent, we have established a level of integrity\. If different, we have a basis for fixing a problem\. In any case, I agree that experience tells who has good data and who doesn't\. In any case, computers don't mean that brains turn off\. Any reading that is unexpected or out of range should be verified \- it is worth a second look\. Ed Hammond              lavanian@vsnl\.net              Sent by:              openehr\-clinical\- To              bounces@openehr\.o For openEHR clinical discussions              rg <openehr\-clinical@openehr\.org>                                                                         cc                                                                                          07/11/2007 06:04 Subject              AM Re: Data quality questions/                                        proposal                                                                                          Please respond to                 For openEHR                  clinical                 discussions              <openehr\-clinical                @openehr\.org>                                                                             Hi folks, Here are my thoughts\. The accuracy of an observation depends on many variables that may or may not be quantifiable\. It could be effected by the man, the machine, the methodology used, personal bias or even deliberate attempts at falsification\. So end of the day what do we, as physicians, do? I can tell you what I do\. I base my trust on experience \- my experience with the results that I get, over the months or years, from a Lab, specialist of paramedic\. and although nothing is absolute, I find that results from certified Labs do turn out to be more consistant than those done by smaller wayside labs\. What I'm stating applies to more than 60% of the population that lives in countries where certification of Labs and regular monitoring is not the norm\. Therefore we could add a field for the Lab's certification\. End of the day, most doctors do not go on Lab results alone; they depend on their own clinical assessment too, like I do\. Warm regards, Dr D Lavanian MBBS, MD Vice President \- Software Division AxSys Healthtech Ltd Mobile: \+91\-9949902800 --- ## Post #28 by @Stef_Verlinden1 Op 15\-jul\-2007, om 17:04 heeft William E Hammond het volgende geschreven: > Stef, > > I would like to share your e\-mail with ACMi to see what responses > we get\. As far as I'm concered that's fine with me\. > It is an interesting proposition\. Question, why not just require > excellent/good measurement? I completely agree with your suggestion\. Thing is that this probably will raise a lot of resistance since everybody then has to implement quality systems before they can share data through an EHR, since they have to show that their data is of good/excellent quality\. My strategy is to show that you can gain/enhance trust by applying a quality system on a few 'easy' cases, where I also involve the patient/ citizen\. Once you can show that trust and quality are indeed crucial factors for truly transmural use and re\-use of medical data the rest will \(have to\) follow\. The latter since citizens will more and more demand good quality healthcare\. I don't know if the Netherlands is an exception internationally, but we have huge local differences \(sometimes a factor 2\-3\) for the normalized 5 year mortality in the treatment of well known and protocoled diseases such as breast cancer\. Only very recently the public is confronted with these figures and already one can see the demand for more transparency and quality\. My firm believe is that that movement will only become stronger in the near future\. Those people will also demand longitudinal/ transmural EHR's with good quality data\. Cheers, Stef --- ## Post #29 by @Sam Thanks Ed I like the idea in your other thread about the likelihood of correctness. What we, and particularly Stef, are up against is hospitals and medical people believing that they have the quality data. White coat effects on 27% of the population (and it seems likely that it is a general effect that is significant in this proportion) means that we may never have the correct BP readings on anyone. Further, general practitioners hate to find high BPs in people they like and know and round down vigorously (as do consultants). This may balance the white coat effect in some patients!! I understand that this is not about quality of measurement - but we do need to accept that devices are accurate when they are. On the whole I predict that they will be considerably better than clinicians from now on - particularly as they can help get around the non-operator confounders like white coat effects. So more grist to Stef's mill - but lets not clutter the record with information that does not actually tell us anything useful. Cheers, Sam William E Hammond wrote: --- ## Post #30 by @Dipak_Kalra Dear Ian, When you say: > there may be a more generic requirement to flag > 'not thought suitable for decision support or secondary uses', so > that the > result \(or entry\) is ignored by machine processing\. after many months of debate and ballot commenting, this is also the decision that was arrived at in finalising EN13606 Part 1 \(EHR Communications Reference Model\): Class ENTRY Attribute uncertainty\_expressed:Boolean Description: This attribute is set to TRUE to advise the EHR Recipient that this ENTRY contains data that indicates some degree of uncertainty, and that care should be taken when using these data within automated processes and systems\. It is a flag to advise the EHR Recipient that the data structure includes some indication of uncertain findings or opinions, and that care needs to be taken when using the data for automated processing It is based on the assumption that uncertainty might be expressed in many ways within an EHR Provider system, including but not limited to coded expressions with or without qualifiers, which might be challenging for an EHR Recipient to detect reliably\. This attribute is a generic way of indicating that the data ought to be critically reviewed \(e\.g\. by a person\) in order to verify if it is suitable for any particular computational process\. Some years of openEHR thinking was provided as input to that international debate, since we had perviously considered a variety of differnt kinds of uncertainty representation and found them largely to be too problematic with regard to consistent usage, or at risk of creating ambiguity when used in parallel with terminology systems that can express uncertainty formally through qualifiers\. With best wishes, Dipak --- ## Post #31 by @system Dear all, When I was a student, I presented a physical examination of a case for the professor\. The professor asked me, "Did you really see the patient eye?" While I presented that the patient's palpebral conjunctivae was not anemic, the professor had the laboratory data that his hemoglobin is 7g/dl\. The professor also said "Don't you think physical examination is irrelevant\. It is true that you feel he is not anemic, but laboratory data is not so\. Our clinical decision was confirmed beyond such discrepancy through many data," Tha subjective data are sometimes irelevant and obvious, but it is just real for the data taker at that time\. Although it is important to evaluate the acuracy of such data, the evaluation is also subjective and obvious\. For clinical decision making, I always remarks the production process of the data, how the data was bringed up for me, who made the data, and when mede the data\. For example, the data made by a resident has less prior than the data made by expert\. --- ## Post #32 by @system Dear all, We can record hard facts: the entity that produced the results and characteristics of that entity (person, organisation, machine, etc) When it comes to indicators to indicate subjective value judgments things become very difficult because they are subjective, very personal. The only wise thing to do is to indicate that some item, some lines, some collection of entries have a question mark and that an expert needs to make his own value judgement. This is, as Dipak indicated, the CEN 13606 way. Gerard --- ## Post #33 by @thomas.beale Gerard Freriks wrote: > Dear all, > > We can record hard facts: > the entity that produced the results and characteristics of that > entity \(person, organisation, machine, etc\) > > When it comes to indicators to indicate subjective value judgments > things become very difficult because they are subjective, very personal\. > > The only wise thing to do is to indicate that some item, some lines, > some collection of entries have a question mark and that an expert > needs to make his own value judgement\. > This is, as Dipak indicated, the CEN 13606 way\. > I have never thought this would work \(Dipak will remember all the debates over the years;\-\) but I am interested to know: if there is a boolean flag that indicates 'some level of uncertainty about this item', what is either a human being or a computer supposed to do with it? If it really means 'the device I used for measuring is not the most accurate, but its ok', then you would probably accept the value; if it means 'this value is a complete guess', then you can't\. But if all you have is the flag, what do you do? Assume the worst case? I think device generated error needs to be recorded in Quantity\.accuracy; I think physician confidence \(in diagnoses, other judegments\) need to be recorded in some quantifiable or ordinal way in the Evaluation\. \- thomas --- ## Post #34 by @Dipak_Kalra Dear Tom, The purpose of the flag is not to tell you anything about the uncertainty within an ENTRY, but only to tell you that it needs to be reviewed as a whole by a person or process capable of reviewing it\. It is not the case that "all you have is a flag", but rather that you have a flag in addition to the contents to warn you that extracting a single part of this ENTRY such as an ELEMENT value through a query might not be safe because some other parts of the ENTRY might be indicating that there is a caveat or caution about the value's interpretation\. The ENTRY contents still need to provide the appropriate details to inform the reviewer, as captured by the original author\. It would be nice if we could be more clever than that, but the complexity of this challenge is such that a single unambiguous and consistently used representation of each kind of uncertainty or caution is not yet feasible \(as we all know\)\. What we found to be appropriate for the standard is to put a label on the box to say "open with caution, don't just drill down blindly and pluck out an isolated value"\. This is by no means a perfect solution but reflects the extent of confidence in the field at present, and what current systems vendors felt could be handled in the near future\. With best wishes, Dipak --- ## Post #35 by @thomas.beale Dipak, I should point out that I am not aiming for any heavy debate of this right now \- it's been done before and is a serious topic\. On the other hand we have all learned more in different areas over the last few years, so it's interesting to bring up a few points and see if anyone's thoughts have changed\. I have mainly questions\. Dipak Kalra wrote: > Dear Tom, > > The purpose of the flag is not to tell you anything about the > uncertainty within an ENTRY, but only to tell you that it needs to be > reviewed as a whole by a person or process capable of reviewing it\. >   this is the question isn't it? What does 'reviewed' mean? If the information is credible \(even if wrong\) how can the reviewer tell, when there is no measure of how wrong? > It is not the case that "all you have is a flag", but rather that you > have a flag in addition to the contents to warn you that extracting a > single part of this ENTRY such as an ELEMENT value through a query > might not be safe because some other parts of the ENTRY might be > indicating that there is a caveat or caution about the value's > interpretation\. The ENTRY contents still need to provide the > appropriate details to inform the reviewer, as captured by the > original author\. >   various questions come to mind:     \* are you saying that any original representation of error etc is       retained in the data?     \* So the flag is really a marker on an Entry to say 'somewhere       buried in here is/are one or more indicators of \(in\)accuracy'?     \* What if all the Quantities have accuracy markers on them \(is this       possible with the CEN QTY data type?\) \- and the accuracies are       e\.g\. \+/\- 5% \(i\.\.e pretty good\) \- do you set the flag or not?     \* What if there were 50 quantities with high accuracy and one of low       accuracy, does the flag get set or not?     \* What if there are differential diagnoses indicating confidence       levels?     \* You wouldn't set the flag on this would you, since the information       is 100% correctly representing what the physician said     \* how hard would it be for software to set this flag?     \* the ultimate question is: does this flag give you any more useful       information than the raw data? \.\.\.or am I missing the point of this altogether? > It would be nice if we could be more clever than that, but the > complexity of this challenge is such that a single unambiguous and > consistently used representation of each kind of uncertainty or > caution is not yet feasible \(as we all know\)\. > > What we found to be appropriate for the standard is to put a label on > the box to say "open with caution, don't just drill down blindly and > pluck out an isolated value"\. how should software react to this? \- thomas --- ## Post #36 by @Stef_Verlinden1 > Gerard Freriks wrote: >> Dear all, >> >> We can record hard facts: >> the entity that produced the results and characteristics of that >> entity \(person, organisation, machine, etc\) >> >> When it comes to indicators to indicate subjective value judgments >> things become very difficult because they are subjective, very >> personal\. >> >> The only wise thing to do is to indicate that some item, some lines, >> some collection of entries have a question mark and that an expert >> needs to make his own value judgement\. >> This is, as Dipak indicated, the CEN 13606 way\. >> > > I have never thought this would work \(Dipak will remember all the > debates over the years;\-\) but I am interested to know: if there is a > boolean flag that indicates 'some level of uncertainty about this > item', > what is either a human being or a computer supposed to do with it? > If it > really means 'the device I used for measuring is not the most > accurate, > but its ok', then you would probably accept the value; if it means > 'this > value is a complete guess', then you can't\. But if all you have is the > flag, what do you do? Assume the worst case? I agree with Gerard that with subjective value measurements things become more \(not very:\-\)\) difficult since those are 'personal/ cultural'\. That doesn't mean that there will be \(at least in one country\) hundred of different criteria for quality\. In most cases there will be one or a few generally accepted set of criteria per 'observation The way I figured how you can deal with this is as follows: \- provide the opportunity to record all relevant objective quality markers in the observation and accompanying archetypes\. \- at \(user\) application level and based on your 'local' standards/ criteria asses whether the data is of sufficient quality\. My additional question is that I want to store that 'local' quality assessment outcome somewhere as well\. Therefore my question is, can we add a generic data quality marker/label, which is adapted for to local situation by specializing that archetype\. Rethinking this, I have to additional thoughts/questions: \- if we asses data quality at application level does one need/want to store the outcome? \- if one wants to store the outcome is it better to use a 'specialized' template for that\. My 'feeling' is that data quality criteria \(especially when they're more widely accepted\) need to be regarded as a standard/protocol and therefore need to be stored in such a 'specialized' archetype/template\. Data quality criteria are not only commonly used but also obliged when for instance developing new drugs \(GLP\) or do clinical testings \(GCP\) for new drug approvals\. Everybody involved use them and it turns out that everybody sticks to the same criteria, since they're actually quite simple: \- is the device used calibrated and maintained properly\. \- Is the device operated by the 'right' person and in the 'right' environment Then of course there are many variations one these themes, depending on the device, which makes it more work to register, but it won't become more complex\. In answer to Thomas, I wouldn't advocate, subjective individual 'assesement/testimonials' of/about data quality without properly defined criteria\. If somebody 'suspects' a device isn't working properly one should take action \(f\.i notify the technical staff and make them re\-calibrate the device\)\. I wouldn't like to be responsible for a patients health if I have to work with data collected from a device that's possibly malfunctioning\. Cheers, Stef --- ## Post #37 by @Stef_Verlinden1 > > Dipak, > > I should point out that I am not aiming for any heavy debate of this > right now \- it's been done before and is a serious topic\. On the other > hand we have all learned more in different areas over the last few > years, so it's interesting to bring up a few points and see if > anyone's > thoughts have changed\. I have mainly questions\. > > Dipak Kalra wrote: >> Dear Tom, >> >> The purpose of the flag is not to tell you anything about the >> uncertainty within an ENTRY, but only to tell you that it needs to be >> reviewed as a whole by a person or process capable of reviewing it\. >> > > this is the question isn't it? What does 'reviewed' mean? If the > information is credible \(even if wrong\) how can the reviewer tell, > when > there is no measure of how wrong? >> It is not the case that "all you have is a flag", but rather that you >> have a flag in addition to the contents to warn you that extracting a >> single part of this ENTRY such as an ELEMENT value through a query >> might not be safe because some other parts of the ENTRY might be >> indicating that there is a caveat or caution about the value's >> interpretation\. The ENTRY contents still need to provide the >> appropriate details to inform the reviewer, as captured by the >> original author\. >> > > various questions come to mind: > >     \* are you saying that any original representation of error etc is >       retained in the data? >     \* So the flag is really a marker on an Entry to say 'somewhere >       buried in here is/are one or more indicators of \(in\)accuracy'? >     \* What if all the Quantities have accuracy markers on them \(is > this >       possible with the CEN QTY data type?\) \- and the accuracies are >       e\.g\. \+/\- 5% \(i\.\.e pretty good\) \- do you set the flag or not? I think one should seperate accuracy from data quality\. Although accuracy is an essential part of data quality, it's not the only parameter\. Accuracy is most of the times dealt with at callibration/ certification level\. A certain device can only be certified/ callibrated if the accuracy is within some tightly defined boundaries\. So a divice is either 'good' or 'bad'\. I'm not aware of 'gray scales' in this respect\. >     \* What if there were 50 quantities with high accuracy and one > of low >       accuracy, does the flag get set or not? The flag is set for each data entry point, so one would get 50 good quality quantities and one poor\. >     \* What if there are differential diagnoses indicating confidence >       levels? >     \* You wouldn't set the flag on this would you, since the > information >       is 100% correctly representing what the physician said >     \* how hard would it be for software to set this flag? >     \* the ultimate question is: does this flag give you any more > useful >       information than the raw data? I agree that this is the ultimate questions\. If all the 'values'' to asses quality are there one can \(re\-\)assess the quality every time needed\. Where I'm struggling with is the following\. If we follow the re\-assess line of thinking, somebody or some agent has to do that every time\. If user have to 'manually' asses the quality of every data entry point it won't happen and than changes are big users won't 'trust' data since its not generated by themselves\. If it has to be computed at user application level, one becomes dependent of the application builder/reseller\. Furthermore everytime data quality criteria change, the builder/reseller has to adapt the application\. So my idea/point is that quality criteria are part of the knowledge domain \(and therefore part of an archetype\) and shouldn't be incorporated in code at application level\. Same could be said about an evaluation: if all the entries on which a evaluation are based are present in the EHR, why would one store the evaluation? Cheers, Stef --- ## Post #38 by @thomas.beale Stef Verlinden wrote: > My additional question is that I want to store that 'local' quality > assessment outcome somewhere as well\. Therefore my question is, can > we add a generic data quality marker/label, which is adapted for to > local situation by specializing that archetype\. Stef, the problem this poses for shared health records is: how to deal with local post facto quality markers when the EHR information \(oriignally from system X\) is sent from system A \(yours\) to system B, C, and say a national summary record? What if it is sent to a number of places? Do we throw away your local quality markers, and let the people at system B, C etc add their own? Do we keep everyone's local annotations, thus accumulating a pile of \(presumably differing\) markers? What if infomration from source X is received by both system A and system B and they both add their own \(different\) quality markers? What if their versions of the data are then sent to another system D? What should system D think? > Rethinking this, I > have to additional thoughts/questions: > \- if we asses data quality at application level does one need/want to > store the outcome? > \- if one wants to store the outcome is it better to use a > 'specialized' template for that\. >   I suspect that a special local database should probably be used\.\.\.\.\. > My 'feeling' is that data quality criteria \(especially when they're > more widely accepted\) need to be regarded as a standard/protocol and > therefore need to be stored in such a 'specialized' archetype/template\. >   I don't think the issue is specialised templates \(you can easily store such information in an optional part of the protocol section of an Entry\); the problem is the idea above that quality markers could be set locally, which in a shared environment means repeated adding of markers and accumulation of markers\. > Data quality criteria are not only commonly used but also obliged > when for instance developing new drugs \(GLP\) or do clinical testings > \(GCP\) for new drug approvals\. Everybody involved use them and it > turns out that everybody sticks to the same criteria, since they're > actually quite simple: > \- is the device used calibrated and maintained properly\. > \- Is the device operated by the 'right' person and in the 'right' > environment > Then of course there are many variations one these themes, depending > on the device, which makes it more work to register, but it won't > become more complex\. > > In answer to Thomas, I wouldn't advocate, subjective individual > 'assesement/testimonials' of/about data quality without properly > defined criteria\. If somebody 'suspects' a device isn't working > properly one should take action \(f\.i notify the technical staff and > make them re\-calibrate the device\)\. I wouldn't like to be responsible > for a patients health if I have to work with data collected from a > device that's possibly malfunctioning\. >   well\.\.\.\.that's the same as now \- it's just that the information wouldn't be as widely shared as in an e\-health world One thing we forgot to mention earlier on, is that in openEHR, attestations can be recorded on versions \- see http://www.openehr.org/uml/release-1.0.1/Browsable/_9_0_76d0249_1109326589721_134411_997Report.html Every time a Version \(of a Composition\) is imported into another system, an attestation can be added to the ImportedVersion wrapper, so it is in theory possible to have an accumulating string of quality flags\. However, it is at a coarse\-grained level, and I am doubtful that it is useful or desriable to require systems to set such markers or to read them\. \- thomas --- ## Post #39 by @Sam Hi The question that is clear in my mind is why don't we do this now...label things with certainty? One difficulty that is immediately apparent is that avoiding risk usually means it is irrelevant. If someone says they had an ulcer, no matter how uncertain I am, will I use a risky therapy? If they say they got nauseated with morphine, will I change my practice at all regardless of certainty? The next issue is rogue measurements - a BP of 200/72 in a healthy person. Even if you have no faith in it, it is still something you need to take into account. When the person is diagnosed with a condition that explains it a few years later, it will look pretty silly if someone has decided it is not reliable and computers have been working on that basis since. In openEHR we have tried to give as much contextual information as we can - with the information provider being able to be stored for each entry - so even if a diagnosis of Crohn's disease gets into the record, if a clinician has insufficient evidence, it is possible to mark that it came from the patient or relative. This is an advance on most current systems. It is akin to a written record "He says he has Crohn's disease". We have modelled 'hard' quality factors in the laboratory archetype - it is probably worth looking at for that reason. It is in the data (optionally) to indicate if there are any quality issues that may affect interpretation. This may be the beginning of something that needs modelling more generally in the future but I would be interested to hear from people as to what are the candidate attributes for this quality issue. We need to do this in a manner that computers can cope....and make sure we don't need any gnomes in the machine. I would like openEHR's moto to be (Getting the Gnomes out of the machine!) Cheers, Sam Thomas Beale wrote: --- ## Post #40 by @Stef_Verlinden1 > Stef Verlinden wrote: >> My additional question is that I want to store that 'local' quality >> assessment outcome somewhere as well\. Therefore my question is, can >> we add a generic data quality marker/label, which is adapted for to >> local situation by specializing that archetype\. > > Stef, the problem this poses for shared health records is: how to deal > with local post facto quality markers when the EHR information > \(oriignally from system X\) is sent from system A \(yours\) to system > B, C, > and say a national summary record? What if it is sent to a number of > places? Do we throw away your local quality markers, and let the > people > at system B, C etc add their own? Do we keep everyone's local > annotations, thus accumulating a pile of \(presumably differing\) > markers? > What if infomration from source X is received by both system A and > system B and they both add their own \(different\) quality markers? What > if their versions of the data are then sent to another system D? What > should system D think? I think I get your point\. The answer could be that we only store the 'hard' quality markers and then compute the data quality locally according to the local criteria everytime we want to view the data\. The thing I'm struggling with is 3 things: \- were to store those local protocols/ criteria\. Do we need to set up a separate system/ database for that or can we store them in a 'localized' archetype/template\. How would one do that with the other protocols we \('re going to\) have to deal with, especially the protocols dealing with work flow/ clinical pathway\. \- we'll have to deal with protocols/ criteria even if you don't want to\. For example the criteria for the diagnoses X may vary locally and change over time \(and this is not an 'exotic' example\)\. So does that mean that you can't store a diagnoses in the EHR and that you have to 're\-evaluate/asses' these everytime you use the EHR from the 'hard' criteria such as symptoms and clinicial and lab findings \(according to your local criteria\)? Thinking about this, is there a place to store the protocol/ criteria by which a certain diagnosis is 'set'? \- I don't know if I understand this correctly, but in order to 'assess the quality of a certain entry performed f\.i\. two years ago, it's necessary to know the exact 'state' of the device at that day\. Is it possible to determine this historic device state 'on the fly' \(i\.e\. isn't that going to take to much time\) or are the other ways to do that? Cheers, Stef --- ## Post #41 by @Stef_Verlinden1 > Hi > > The question that is clear in my mind is why don't we do this > now\.\.\.label things with certainty? One difficulty that is > immediately apparent is that avoiding risk usually means it is > irrelevant\. If someone says they had an ulcer, no matter how > uncertain I am, will I use a risky therapy? If they say they got > nauseated with morphine, will I change my practice at all > regardless of certainty? > > The next issue is rogue measurements \- a BP of 200/72 in a healthy > person\. Even if you have no faith in it, it is still something you > need to take into account\. When the person is diagnosed with a > condition that explains it a few years later, it will look pretty > silly if someone has decided it is not reliable and computers have > been working on that basis since\. > > In openEHR we have tried to give as much contextual information as > we can \- with the information provider being able to be stored for > each entry \- so even if a diagnosis of Crohn's disease gets into > the record, if a clinician has insufficient evidence, it is > possible to mark that it came from the patient or relative\. This is > an advance on most current systems\. It is akin to a written record > "He says he has Crohn's disease"\. > > We have modelled 'hard' quality factors in the laboratory archetype > \- it is probably worth looking at for that reason\. It is in the > data \(optionally\) to indicate if there are any quality issues that > may affect interpretation\. This may be the beginning of something > that needs modelling more generally in the future but I would be > interested to hear from people as to what are the candidate > attributes for this quality issue\. The thing I"m trying to address is about these 'hard' quality factors, not the 'soft' ones\. From my point of view we need to record those 'hard' factors in order to be able to compute/compare them against quality criteria in order to create 'trust', but the question is where? That's why I thought of a device AT\. In that situation device related quality data has to be recorded only once, while it can be easily linked to 'every' measurement performed with that device\. For every data entry the 'state' of the device an be 'known' and be taken into account\. If we can do the same thing with a 'cluster' as you suggested, or otherwise, it's fine with me\. > We need to do this in a manner that computers can cope\.\.\.\.and make > sure we don't need any gnomes in the machine\. I would like > openEHR's moto to be \(Getting the Gnomes out of the machine\!\) I completely agree\. My other point is: if a computer copes with this quality issue there are basically 2 possible outcome: data is of sufficient quality or data is of insufficient quality \(of course this will depend on the criteria one needs to set on forehand\)\. My other question is where do we store this 'outcome' of such a quality assessment\. My suggestion is to add to the protocol a 'data quality field' for that purpose\. Furthermore to embed/add the criteria in the ontology section, so these are clear\. Since I'm aware that these criteria could vary locally these criteria only should be added in a local specialization of this archetype/template \(see previous reply\)\. The other day I tried to sent another post which contained an 'example' AT in which this field was included\. Unfortunately it bounced since it was too large\. So here I'll only provide the relevant parts of that example AT\. Under the protocol part a data quality element \(at1019\) is added which can be used to qualify the data entry\. Under ontology the criteria for assessing data quality are given \(at1020, at 1024, at 1025, at1026, at1027\)\. protocol matches \{       ITEM\_TREE\[at0011\] matches \{ \-\- List structure         items cardinality matches \{0\.\.\*; ordered\} matches \{           ELEMENT\[at0013\] occurrences matches \{0\.\.1\} matches \{ \-\- Cuff size             value matches \{               DV\_CODED\_TEXT matches \{                 defining\_code matches \{                   \[local::                   at0015, \-\- Adult                   at0016, \-\- Wide adult                   at0017, \-\- Paediatric                   at1008, \-\- Thigh                   at1009\] \-\- Neonatal                 \}               \}             \}           \}           ELEMENT\[at1019\] occurrences matches \{0\.\.1\} matches \{ \-\- Data quality             value matches \{               DV\_CODED\_TEXT matches \{                 defining\_code matches \{                   \[local::                   at1020, \-\- Excellent                   at1024, \-\- Good                   at1025, \-\- Fair                   at1026, \-\- Poor                   at1027\] \-\- Unknown                 \}               \}             \}           \} >         \["at1019"\] = <           description = <"\*Score for the quality of the obtained data during this session \(en\)">           text = <"\*Data quality\(en\)">         >         \["at1020"\] = <           description = <"\*data measured/obtained by a qualified person, with a certified instrument/device thats calibrated against a golden standard, the measurement error is within a tight bandwidth \(<5%\) , the validity duration of the calibration isnt expired, maintained on time and by qualified personal\(en\)">           text = <"\*Excellent\(en\)">         >         \["at1024"\] = <           description = <"\*data measured/obtained by a qualified person \(this can also be a properly trained citizen\), with a certified instrument/device thats self calibrating, the measurement error is within a tight bandwidth \(<5%\) , the machine isnt broke and functioning well\(en\)">           text = <"\*Good\(en\)">         >         \["at1025"\] = <           description = <"\*data measured/obtained by a qualified person \(this can also be a properly trained citizen\), with a certified instrument/device thats self calibrating, the measurement error isnt within a tight bandwidth \(CE marking alone allows measurement errors >5%, the machine is not broke and functioning well\(en\)">           text = <"\*Fair\(en\)">         >         \["at1026"\] = <           description = <"\*in all the other situations\(en\)">           text = <"\*Poor\(en\)">         >         \["at1027"\] = <           description = <"\*it is unknown how the data was gathered\(en\)">           text = <"\*Unknown\(en\)"> Cheers, Stef --- ## Post #42 by @system > I think I get your point. The answer could be that we only store the > > 'hard' quality markers and then compute the data quality locally > > according to the local criteria everytime we want to view the data. This makes sense. "hard quality markers" are determinants for the subjective interpretation. These locally interpreted "hard quality markers" end up in an annotation expressing the local belief at that point in time. GF --- ## Post #43 by @thomas.beale Stef Verlinden wrote: > > The thing I'm struggling with is 3 things: > \- were to store those local protocols/ criteria\. Do we need to set up > a separate system/ database for that or can we store them in a > 'localized' archetype/template\. How would one do that with the other > protocols we \('re going to\) have to deal with, especially the > protocols dealing with work flow/ clinical pathway\. >   My feeling based on what you have said is still that the markers should be in the demographic system, and can be attached to any provider organisation, device, software or team that generates information that you consume\. > \- we'll have to deal with protocols/ criteria even if you don't want > to\. For example the criteria for the diagnoses X may vary locally and > change over time \(and this is not an 'exotic' example\)\. I wouldn't have thought the variation was that great, but in any event, the openEHR architecture doesn't care: if physician A is recording numerous diagnoses of hypertension for BP = 160, and physician B isn't, then they are clearly using different guidelines \(or maybe none\)\. Either way, there is the pace in the information structure to record which guideline, or other clinical reasoning, and also links can be set to point to the previously recorded observations that indicate this diagnosis\. So if a 3rd system is processing these records, and sees ICD\-coded diagnoses for hypertension, it will be able to discover which ones are 'real' according to the guideline it knows about\. > So does that > mean that you can't store a diagnoses in the EHR and that you have to > 're\-evaluate/asses' these everytime you use the EHR from the 'hard' > criteria such as symptoms and clinicial and lab findings \(according > to your local criteria\)? Thinking about this, is there a place to > store the protocol/ criteria by which a certain diagnosis is 'set'? >   This would normally be in a guideline that would be referenced from the EHR from the CARE\_ENTRY\.guideline\_id \(see http://www.openehr.org/uml/release-1.0.1/Browsable/_9_0_76d0249_1109249648736_872559_12384Report.html), or an informal comment can be put in the protocol \(since a guideline is essentially about 'how' you came to the result recorded in the main part of the Entry\)\. > \- I don't know if I understand this correctly, but in order to > 'assess the quality of a certain entry performed f\.i\. two years ago, > it's necessary to know the exact 'state' of the device at that day\. > Is it possible to determine this historic device state 'on the > fly' \(i\.e\. isn't that going to take to much time\) or are the other > ways to do that? >   this is achievable by using a demographic service based on the openEHR demographic model \(http://www.openehr.org/uml/release-1.0.1/Browsable/_9_5_76d0249_1118674798473_6021_0Report.html), and implementing the versioning semantics \(same as in the EHR \- see here http://www.openehr.org/uml/release-1.0.1/Browsable/_9_0_76d0249_1109326589721_134411_997Report.html). Of course in the deployed system, there must be someone responsible for actually updating device quality / calibration markers ;\-\) I can see some very valid concerns in all of the discussion so far, but I think we need to walk before we can run\.\.\. \- thomas --- ## Post #44 by @Stef_Verlinden1 OK and thanks. Let's leave it here for now and start walking:-) I'd love to meet in person to discus some future directions. Also I'd love to come together with a couple of skilled clinical people to learn, share experiences and see what the general idea is how we should deal with these type of issues. So if people are interested let's see if we can organize something. I'm of now for Holidays. Cheers, Stef --- ## Post #45 by @Sam Hi Stef For other people's benefit, Stef is talking about taking the EHR into people's homes so he wants to have some data that will give us some idea of quality. I think we would do better to have the data harder than this. The first is the issue of the training of the person doing the measurement - this is perhaps important for people measuring their own BP if they want to put it into the EHR - this should be recorded somewhere that they have been trained. We need to recognise that the place of measurement is also important - and I would put this in the protocol as well as the actual data might be added later when at the surgery. As for the accuracy - I believe this has to come from the device with the measurement or it is unlikely to be used or reliable. We should know what are the IDs of devices (perhaps Manufacturer and Model Number) and I would think the last time it was calibrated would be useful if required. Your classification (in the ADL below) would have to be calculated based on the data, I would think, if it was going to be reliable. I do not have a problem as long as the attributes are not mandatory in the archetype - rather set in the template in the use environment. The data that we are going to get from the Continua consortium will be useful. You mentioned people checking their INR - I expect calibration will be essential in that case - also BSLs. So perhaps we should have a generic cluster for the type of machine and date of calibration that we can add to all protocols where this is an issue. Send me the ADL and I will have a go.... Cheers, Sam Stef Verlinden wrote: --- ## Post #46 by @Stef_Verlinden1 > Hi Stef > > For other people's benefit, Stef is talking about taking the EHR into people's homes so he wants to have some data that will give us some idea of quality. > > I think we would do better to have the data harder than this. The first is the issue of the training of the person doing the measurement - this is perhaps important for people measuring their own BP if they want to put it into the EHR - this should be recorded somewhere that they have been trained. We need to recognise that the place of measurement is also important - and I would put this in the protocol as well as the actual data might be added later when at the surgery. I completely agree, that's why we're including an e-learning module in the apllication. Hopefully I'll be able to show you a dummy/ mock-up shortly after my holidays. Another option is that someone is trained by his GP/healthcare provider. This is the reason why I think we need to add these 'capabilties' to the demogrpahic AT attached to the role of patient/citizen. > As for the accuracy - I believe this has to come from the device with the measurement or it is unlikely to be used or reliable. We should know what are the IDs of devices (perhaps Manufacturer and Model Number) and I would think the last time it was calibrated would be useful if required. Or from a certifying instance. Take for example (home) bloodpressure measurement devices: The British Society of Hypertension tests and certifies these devices and they pass based on very strict criteria including accuracy. Once certified and maintained properly one can rely on the accuracy. In that case one would like to add the 'certificate' to the AT. For non certified device one might need to establish accuracy themselves and store this somewhere (in a device AT/ cluster) > Your classification (in the ADL below) would have to be calculated based on the data, I would think, if it was going to be reliable. I do not have a problem as long as the attributes are not mandatory in the archetype - rather set in the template in the use environment. The data that we are going to get from the Continua consortium will be useful. You mentioned people checking their INR - I expect calibration will be essential in that case - absolutely > also BSLs. So perhaps we should have a generic cluster for the type of machine and date of calibration that we can add to all protocols where this is an issue. > > Send me the ADL and I will have a go.... I'll send it directly since it's too large to attach. I'll still need some help with the 'cluster' part for the device Cheers Stef --- **Canonical:** https://discourse.openehr.org/t/data-quality-questions-proposal/14665 **Original content:** https://discourse.openehr.org/t/data-quality-questions-proposal/14665