Flavour of null

Technical (archive)
21

Dear Roger and Thomas,

We have looked extensively at Multivalued logic for quantitating uncertainty. It turns out that most folks in that world have taking 0 false and one true with a number of discrete, usually equally spaced values in between for uncertainty.

After a longwinded go around with a Prof of Philosophical Logic at Princeton (Dr. Graham) We determined that there at least three reproducible types of uncertainty (with good inter-rater reliability) and ~ seven semantic categories.

The types are Probable (our guess is around 85% true +/- 5%) and Unlikely (our guess is around 15% true +/- 5%) or Just as likely as not (again our guess is around 50% +/- 15%). These number come from the average PPV of the evidence when a physician "Makes a diagnosis" and NPV when a physician rules one out.

Other distinctions are less reproducible. When taken together most clinicians would say that Probable is stronger than Likely, however the assignment to actual cases is not in our experience reproducible between knowledgeable reviewers.

I suggest that you first code (as we do) True, False or Uncertain. Then qualify Uncertain with a semantic type indicating strength. This allows a model that can grow with our ability to represent more closely evidence based medicine.

Warm regards,

Peter

Peter L. Elkin, MD
Professor of Medicine
Director, Laboratory of Biomedical Informatics
Department of Internal Medicine
Mayo Clinic, College of Medicine
Mayo Clinic, Rochester
(507) 284-1551
Fax: (507) 284-5370

22

Elkin, Peter L., M.D. wrote:

Dear Roger and Thomas,

We have looked extensively at Multivalued logic for quantitating uncertainty. It turns out that most folks in that world have taking 0 false and one true with a number of discrete, usually equally spaced values in between for uncertainty.

After a longwinded go around with a Prof of Philosophical Logic at Princeton (Dr. Graham) We determined that there at least three reproducible types of uncertainty (with good inter-rater reliability) and ~ seven semantic categories.

The types are Probable (our guess is around 85% true +/- 5%) and Unlikely (our guess is around 15% true +/- 5%) or Just as likely as not (again our guess is around 50% +/- 15%). These number come from the average PPV of the evidence when a physician "Makes a diagnosis" and NPV when a physician rules one out.

[with appropriate excuses in advance for my engineer's view of clinical things;-]

I presume that these values (which seem entirely reasonable to me) were obtained by a statistical study of clinicians' notes? Or interviews? But the problem we are always concerned with is: what does one clinician mean when s/he says "probable rheumatoid arthritis"? We can't assume it can be translted into 85% +/- 5% can we? The particular physician who said it might habitually and unconsciously put "probable" all over the place, when they should really put "possible". Sam's point of view so far has been: make them enter a number (prompt = "% probability of being true" or similar). I know that doesn't address the perfectly reasonable need to allow clinical people to write "probable", "possible" etc, so maybe it's not a long term answer.

But let's just consider what doing clinical medicine is about: it's just scientific problem-solving. The goal is to fix a problem (with the patient); the method is to iteratively gather information until a conclusion (diag = Rh Arthritis) can be drawn or a decision can be made (commence ibuprofen). Fixing a problem may involve many repetitions of this until the problem is fixed. Now, whenever (lack of ) confidence or uncertainty occurs, it means that we don't have enough information to make a decision or draw a conclusion, at least not the next one in the chain. But we do have an indication of what to do next - usually gather more information.

So perhaps the way we view words like "possible", "likely", "probable" should be as motivators to perform more actions to reduce the uncertainty. If a doctor writes "possible malaria" re: a patient just back from a holiday vietnam, with heavy flu-like symptoms, the obvious implication is to do the appropriate microscopy & other diagnostic procedures for malaria, to rule it out or otherwise. For most diseases, a diagnostic algorithm or guideline is available, and the physician having used a word implying uncertainty just means that the diagnostic process is currently at some interior node of such a guideline tree. The key question is probably _which_ of the possible next steps to rule out /rule in one of the differential diagnoses to do in which order - i.e. which is cheapest, fastest, most relevant to patient health etc.

So my question to clinicians is this: doesn't a note containing "possible X", "likely Y" really imply a differential diagnosis, even if only one of the possibilities is actually noted? If so, it may not matter what the level of uncertainty is so much; what matters (among other things) is the severity of the consequences of any of the possible branches of the differential diagnosis. E.g. if one of the implied or noted branches of a differential diagnosis for a patient presenting fever is malaria, presumably both patient and doctor want to discount it as fast as possible, and pursue the appropriate steps to do so. But if none of the branches is life-threatening, reasonable action may be "wait 12 hours" and re-assess. The very common situation of infant presenting with fever must present such a quandary daily.

I'm wondering if there is a meta-algorithm of some sort lurking behind the scenes, which takes account of uncertainty in a note, and also severity of non-discounted possibilities, as a way of deciding what to do next. There is undoubtedly published work on this...

thoughts?

- thomas beale

23

Thomas Beale wrote:

I'm wondering if there is a meta-algorithm of some sort lurking behind
the scenes, which takes account of uncertainty in a note, and also
severity of non-discounted possibilities, as a way of deciding what to
do next. There is undoubtedly published work on this...

This is a very brief but reasonable introduction to Bayesian probability
(which includes calculation of utility), which is what I think you are
grasping at: http://en.wikipedia.org/wiki/Bayesian_probability

Tim C

24

Tim Churches wrote:

Thomas Beale wrote:

I'm wondering if there is a meta-algorithm of some sort lurking behind
the scenes, which takes account of uncertainty in a note, and also
severity of non-discounted possibilities, as a way of deciding what to
do next. There is undoubtedly published work on this...
   
This is a very brief but reasonable introduction to Bayesian probability
(which includes calculation of utility), which is what I think you are
grasping at: http://en.wikipedia.org/wiki/Bayesian_probability

Hi Tim,
and there are quite a few decision support products based on Bayesian logic as well. But I wonder if they have been applied to the problem of determining next best steps based not just on clinical data so far, but also cost, duration, and perceived severity of consequences of not doing something. And I think that Bayesian products should take as inputs only weightings proven by population studies, whereas physician belief is often supported by informal but often qutie accurate personal experience (i.e. experience of the patient population of the practice).

In any case, can we argue that there is no point caring about any finer gradations of true/false than true/false/maybe, as Peter Elkin has said they are doing at Mayo?

- thomas

25

I agree that clinical diagnosis is about problem solving, although it's
'scientific' credentials are often rather weak.

From a scientific point of view, one's 'confidence' in a hypothesis, e.g. a

diagnosis, does not correspond to a quantifable 'likelihood of truth' (as
Popper has insisted for years) but to the set of hypotheses all of which
account for the facts observed so far.
Scientific problem solving seeks optimal ways of making observations that could
refute as many members of this set as possible.
Certainty is reached when the set of valid hypotheses is singular. Note that
this is certainty is not equivalent to truth, since the 'true' hypothesis may
be one that one has not, yet, been able to articulate; hence Kuhn's 'paradigm
shift'.
In clinical medicine, 'Clinical Practice Guidelines' (CPGs) provide what
practitioners have discovered, so far, to be optimal strategies for diagostic
testing in certain areas.

Quoting Thomas Beale <thomas@deepthought.com.au>:

26

Dear Thomas,

I think we need clinicians to be more precise in these declarations. If we begin to train clinicians that Probable should mean ~85% probability +/- 5% then we will move closer to stability.

Although the goal of reducing uncertainty is in general laudible there are some problems that crop up first clinicians are usually only about 90% sure by evidence when they "make a diagnosis" if looked at from an EBM perspective. Also the path to reduction of uncertainty takes into account what prior data is available, the risk benefit ratio of obtaining each piece of data, and patient preference.

Interesting but not easy.

Peter
Peter L. Elkin, MD
Professor of Medicine
Mayo Clinic College of Medicine

27

-1- Almost never a diagnosis is 100% certain.
-2- Almost always a test result has uncertainty attached to it
-3- Many times a conclusion is reached based on many uncertain and conflicting facts
-4- Quite often a condition, a diagnosis, is assumed that gives rise to a treatment. Not indicating that the patient is suffering from this condition but using treatment as a test procedure. Doing nothing is such a test procedure.

Eric Wulff (from Danmark) published philisophical texts about health care and these topics.

gerard

-- <private> --
Gerard Freriks, arts
Huigsloterdijk 378
2158 LR Buitenkaag
The Netherlands

+31 252 544896
+31 654 792800

28

Hi all.
This is an important topic. Here are some references / pointers for those who wish to read more:

"Decision making in health and medicine. Integrating evidence and values" Myriam Hunink and Paul Glasziou Cambridge university press (ISBN 0 521 77029 7)

Society for Medical Decision Making: http://www.smdm.org/

I also recommend journal articles written by Wimla L Patel (Colombia university, New York), for instance:
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=11418539
(A primer on aspects of cognition for medical informatics)

regards
arild Faxvaag

På 22. apr. 2005 kl. 07.42 skrev Gerard Freriks:

29

Hi Arild,

Another site is the MIT Group on Clinical Decision Making: [ http://medg.lcs.mit.edu/ ].
"... a research group dedicated to exploring and furthering the application of technology and artificial intelligence to clinical situations. Because of the vital and crucial nature of medical practice, and the need for accurate and timely information to support clinical decisions, the group is also focused on the gathering, availability, security and use of medical information throughout the human "life cycle" and beyond ..."

Unfortunately Patient decision-making receives less emphasis and studies seem to miss some
fundamental factors (e.g., it is private)
[ http://www.ahrq.gov/research/rtisumm.htm ]

Regards!

-Thomas Clark

Arild Faxvaag wrote:

30

-1- Almost never a diagnosis is 100% certain.

Most of healthcare professionnal I met consider that the term "Diagnosis" means "working hypothesis."

Regards,

-- Patrick Lefebvre ------------- ( plefebv@free.fr ) ----------------------
    "Ce que j'écris n'engage que moi, et ce jusqu'à ma prochaine idée."

31

We could say that physicians infer diagnostic hypotheses based on

  • knowledge of the tentative underlying disease,

  • the patients subjective experiences

  • phenomena registered in the patients body

In any case it is a subjective statement and is a professional opinion based on more (lab results, x-rays) or less (patient history) objective data.

Phrases such as “cannot be exluded” might be due to", “probably”, “definitely”, “beyond doubt” are statements of probability of the inferrence being correct (and what to do next).

Inferrences expressed in the subjective statements documenting the treatment of the patient.

Can one say that diagnoses belong to the class of statements whereas the disease itself belong to the class of natural phenomena?

Disease is an abstraction of reality that for the moment, for the next decision is considered to represent the reality about the health of the patient.
Diagnosis is the professional but subjective opinion about a disease of a patient.

There is a continuum:
Real pathological, fysiologiscal phenomena in a patient.
Certain manifestations of these phenomena.
That are (or are not) experienecd by the patient of an other person.
The arrousal of distress, anxiety, etc, triggering a visit to a physisian.
What is said (or not) about the manifestations of the phenomena during the visit.
And how it is said.
How it is measured and documented.
What is understood of what was said or measured about the manifestations of the phenomena.
How all this was mached to the state-of-the art knowledge, or interpreted in the context of a limited amount of available knowledge.
What was recorded about all these steps above.
How the same person (or others) interpret the recorded ‘facts’ at a later stage???

So what do we record in an EHR?
And what do we interpret readingan EHR?
Then …
What is certain?
And what is uncertain?
Certain or uncertain in what domain, in what line above, at what level of the whole described continuum?

In ±25% of the extremely wel researched patients in one University Hospital we not diagnosed correctly during their life time.
As could be concluded after an autopsy.
So what do we really know about disease and complaints?

What is certainty?
What is it refering to?

Do we understand this mine field well enough?

The diagnosis establishes a relation between the subjective experiences / phenomena and the disease that induces those symptoms and findings.

Example:

Experiences and phenomena: Pain in the wrist joints, feeling of joint stiffness, joint tenderness, joint swelling, elevated sedimentation rate.

Diagnostic inferrence: Rheumatoid arthritis.

Relation: Might be induced by/due to

Can statements of probability be considered statements regarding the strength of these relations??

This is what they are at best.

32

In ±25% of the extremely wel researched patients in one University Hospital we not diagnosed correctly during their life time.
As could be concluded after an autopsy.
So what do we really know about disease and complaints?
What is certainty?
What is it refering to?
Do we understand this mine field well enough?

I recommend this book: "Decision making in health and medicine Integrating evidence and values" by M Hunink and P Glasziou.

Speaking with experience from rheumatology and general practice in Norway, I agree that the state of the diagnosis in medical records is lousy. Just a few points:
- as stated by Gerard, physicians very often come up with the wrong diagnosis, sometimes with fatal consequences.
- how strong the physician believes in the diagnostic hypothesis is not stated explicitly.
-- whether this certainty/probability is above or below the test-treat threshold (depends among others on the expected utility of the treatment) is not stated.
-- whether this certainty/probability is above or below the no treat (wait) - test threshold
- too many resources are spent on excuding differential diagnoses whose (pretest) probabilities already are below the no treat - test threshold
--this to maintain the trust from the patient / avoid the risk of litigation.

The actions of health care personnel have norms. What entity a "diagnosis" is can also be evaluated according to what the norms say it _should_ be. The diagnosis _ought_ to be an inferrence drawn from medical knowledge and information which stems from the patient.

It is not the underlying disease but the physician's inferred diagnosis that form the basis of all health interventions. Because of this, it deserves to be represented as more than a subjective statement.

In an EHR system, it should be possible to link the diagnosis statement with its underlying premises. It should also be possible to link the diagnosis to the set of plans/actions that follows as a consequence. This would lay the foundation for EHR systems that visualizes the consequences of physician's actions in a much better ways than in systems of today.

regards
Arild fax