# Byte Order Marks

**URL:** <https://discourse.openehr.org/t/byte-order-marks/14842>\
**Category:** Technical (archive)\
**Created:** [25 October 2008 09:27 UTC](https://discourse.openehr.org/t/byte-order-marks/14842 "2008-10-25T09:27:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adam\_Flinton](https://discourse.openehr.org/letter_avatar_proxy/v4/letter/a/b782af/32.png) [@Adam\_Flinton](https://discourse.openehr.org/u/Adam_Flinton)\
**Post date:** [25 October 2008 09:27 UTC](https://discourse.openehr.org/t/byte-order-marks/14842/1 "2008-10-25T09:27:04Z")

</div>

Byte Order Marks

What is the default CharSet for OpenEHR ADL?

ASCII? UTF-8?

I ask because ADL itself does not anywhere declare a character set & we  
have had a number of adl files which have failed (either to be opened or  
to be transformed into XML) & in each occasion the reason has been the  
presence of a byte order mark (hex bytes EF BB BF) e.g.

Exception in thread "main" se.acode.openehr.parser.TokenMgrError:  
Lexical error at line 1, column 1. Encountered: "\u00ef" (239), after : ""  
&nbsp;&nbsp;&nbsp;&nbsp;at  
se.acode.openehr.parser.ADLParserTokenManager.getNextToken(ADLParserTokenManager.java:27554)

Equally if a text editor opens an ADL, assumes UTF-8 & puts on a BOM  
then the Archetype editor dies in addition to the Java ADL parser & the  
Windows ADL2XML converter.

Our standard for XML is UTF-8 & I am wondering that if the std is in ADL  
ASCII then how does/will adl support extended charater sets?

e.g. in one of our adl there is some Dutch including

"Een patiënt in rolstoel moet zonder hulp met hoeken en deuren kunnen  
omgaan" where "patiënt" is mis-rendered (though given mail clients are  
pretty good this will probably be correctly rendered)

With other tooling we always had a problem wrt people pasting in content  
from other tools (mostly Word) which had a non-UTF-8 codeset.  
Equally we have a fair amount of existing content which might also be  
pasted into the ADL files via the Archetype editor.  
How does the Archetype editor deal with non-ASCII chars if they are  
pasted in?

Is there a possible loss of fidelity when converting between ADL (in  
ASCII) and XML in UTF8?

So...

A) In general what are the stds etc for Char Sets & ADL?  
B) The various parsers etc should not blow up upon running into a std  
Byte Order Mark.  
C) Would setting the ADL to a non BOM UTF (e.g. UTF-16LE) be OK?

Right now I can simply clean up the adl by running each one through a  
CharsetEncoder/Decoder however....I would prefer this to be fixed at source.

Adam

---

<div class="post-metadata">

**Author:** ![thomas.beale](https://discourse.openehr.org/user_avatar/discourse.openehr.org/thomas.beale/32/35_2.png) [@thomas.beale](https://discourse.openehr.org/u/thomas.beale)\
**Post date:** [25 October 2008 11:05 UTC](https://discourse.openehr.org/t/byte-order-marks/14842/2 "2008-10-25T11:05:41Z")

</div>

Adam Flinton wrote:

> Byte Order Marks
> 
> What is the default CharSet for OpenEHR ADL?
> 
> ASCII? UTF-8?  
> &nbsp;&nbsp;  
> Hi Adam,

UTF-8 is the preferred. See section 3 of  
[http://www.openehr.org/releases/1.0.1/architecture/am/adl.pdf](http://www.openehr.org/releases/1.0.1/architecture/am/adl.pdf)

> I ask because ADL itself does not anywhere declare a character set & we  
> have had a number of adl files which have failed (either to be opened or  
> to be transformed into XML) & in each occasion the reason has been the  
> presence of a byte order mark (hex bytes EF BB BF) e.g.
> 
> Exception in thread "main" se.acode.openehr.parser.TokenMgrError:  
> Lexical error at line 1, column 1. Encountered: "\u00ef" (239), after : ""  
> &nbsp;&nbsp;&nbsp;&nbsp;at  
> se.acode.openehr.parser.ADLParserTokenManager.getNextToken(ADLParserTokenManager.java:27554)
> 
> Equally if a text editor opens an ADL, assumes UTF-8 & puts on a BOM  
> then the Archetype editor dies in addition to the Java ADL parser & the  
> Windows ADL2XML converter.  
> &nbsp;&nbsp;  
> BOMs should not be used in UTF-8 files, they are only for UTF-16 files.  
> Lots of broken programs unfortunately seem to add a BOM for UTF-8, but  
> it breaks things on unix, or unix-like environments. The ADL parser in  
> the ADL Workbench detects BOMs and ignores them. You can try the  
> unicode/family\_history archetype in the dev/test area on SVN in the ADL  
> Workbench - it is in farsi, to see unicode working. It works in all  
> languages we have tested.

If you really want proof (😉 you can see how the ADL parser (used  
inside the Archetype Editor and ADL workbench) does its character  
matching - see  
[http://www.openehr.org/svn/ref\_impl\_eiffel/BRANCHES/specialisation/libraries/common\_libs/src/structures/syntax/dadl/parser/dadl\_scanner.l](http://www.openehr.org/svn/ref_impl_eiffel/BRANCHES/specialisation/libraries/common_libs/src/structures/syntax/dadl/parser/dadl_scanner.l)  
and  
[http://www.openehr.org/svn/ref\_impl\_eiffel/BRANCHES/specialisation/components/adl\_parser/src/syntax/cadl/parser/cadl\_scanner.l](http://www.openehr.org/svn/ref_impl_eiffel/BRANCHES/specialisation/components/adl_parser/src/syntax/cadl/parser/cadl_scanner.l)

these are the dADL and cADL lexers respectively. They parse all strings  
in a UTF-8 aware way. In addition, you can see here where the BOMs are  
removed from ADL files, if found -  
[http://www.openehr.org/svn/ref\_impl\_eiffel/BRANCHES/specialisation/libraries/common\_libs/src/file\_system/file\_context.e](http://www.openehr.org/svn/ref_impl_eiffel/BRANCHES/specialisation/libraries/common_libs/src/file_system/file_context.e)

> Our standard for XML is UTF-8 & I am wondering that if the std is in ADL  
> ASCII then how does/will adl support extended charater sets?
> 
> e.g. in one of our adl there is some Dutch including
> 
> "Een patiënt in rolstoel moet zonder hulp met hoeken en deuren kunnen  
> omgaan" where "patiënt" is mis-rendered (though given mail clients are  
> pretty good this will probably be correctly rendered)  
> &nbsp;&nbsp;  
> See above, there is no problem in ADL files - they support UTF-8 and  
> have been tested extensively. If there are problems in other tools, we  
> need to look carefully at the actions that lead to the problems. I can't  
> answer off the top of my head whether there are cut and paste problems  
> in the Archetype Editor, but I don't believe there should be any  
> problems in its ADL serialisation. Perhaps there are inconsistencies in  
> the XML serialiser - it is a separate piece of code. We should certainly  
> get it fixed at source if there are any problems, but I am sure they  
> will be in the code, not the archetype files themselves.

- thomas beale

---

<div class="post-metadata">

**Author:** ![Peter\_Gummer1](https://discourse.openehr.org/letter_avatar_proxy/v4/letter/p/258eb7/32.png) [@Peter\_Gummer1](https://discourse.openehr.org/u/Peter_Gummer1)\
**Post date:** [25 October 2008 22:55 UTC](https://discourse.openehr.org/t/byte-order-marks/14842/3 "2008-10-25T22:55:12Z")

</div>

Adam Flinton wrote:

> Equally if a text editor opens an ADL, assumes UTF-8 & puts on a BOM  
> then the Archetype editor dies ...  
> &nbsp;&nbsp;  
> I assume you mean the Java Archetype Editor, Adam. The Ocean Archetype  
> Editor accepts ADL files with or without the BOM.

There are pros and cons whether tools should put a BOM at the start of  
ADL files.

\* As Thomas pointed out, tools that are not Unicode-aware may blow up if  
the BOM is present.

\* On the other hand, if you omit the BOM then Unicode-aware tools have a  
big problem when they open a file. What encoding should they assume?  
Some tools like Windows Notepad seem to be very clever at figuring it  
out, but others that I have tried in the past (Visual Studio 2005, Vim  
6.4 and Mac TextEdit) misinterpret BOM-less UTF-8 files as Latin-1.

- Peter

---

<div class="post-metadata">

**Author:** ![system](https://discourse.openehr.org/uploads/default/original/2X/f/f0a1dedb20c42747bddcafd6c7df9db5f34f003c.svg) [@system](https://discourse.openehr.org/u/system)\
**Post date:** [27 October 2008 07:03 UTC](https://discourse.openehr.org/t/byte-order-marks/14842/4 "2008-10-27T07:03:50Z")

</div>

Peter Gummer wrote:

> Adam Flinton wrote:  
> &nbsp;&nbsp;
> 
> > Equally if a text editor opens an ADL, assumes UTF-8 & puts on a BOM  
> > then the Archetype editor dies ...  
> > &nbsp;&nbsp;&nbsp;&nbsp;  
> > I assume you mean the Java Archetype Editor, Adam. The Ocean Archetype  
> > Editor accepts ADL files with or without the BOM.  
> > &nbsp;&nbsp;

I am pretty sure that recent versions of the Java Parser won't mind the  
BOM being present.  
The Java Archetype Editor may still use an old version of the parser?

> There are pros and cons whether tools should put a BOM at the start of  
> ADL files.
> 
> \* As Thomas pointed out, tools that are not Unicode-aware may blow up if  
> the BOM is present.
> 
> \* On the other hand, if you omit the BOM then Unicode-aware tools have a  
> big problem when they open a file. What encoding should they assume?  
> Some tools like Windows Notepad seem to be very clever at figuring it  
> out, but others that I have tried in the past (Visual Studio 2005, Vim  
> 6.4 and Mac TextEdit) misinterpret BOM-less UTF-8 files as Latin-1.  
> &nbsp;&nbsp;

That's why my assumption till now was that the BOM is optional [but not  
forbiddedn] for UTF-8 - and also is of some additional value to  
differentiate between UTF-8 and ISO-8859-1 (as long as you assume that a  
text doesn't start with  - the BOM in ISO, one can safely  
differentiate between the two without problems)

Sebastian

---

<div class="post-metadata">

**Author:** ![Adam\_Flinton](https://discourse.openehr.org/letter_avatar_proxy/v4/letter/a/b782af/32.png) [@Adam\_Flinton](https://discourse.openehr.org/u/Adam_Flinton)\
**Post date:** [3 November 2008 13:25 UTC](https://discourse.openehr.org/t/byte-order-marks/14842/5 "2008-11-03T13:25:14Z")

</div>

> See above, there is no problem in ADL files - they support UTF-8 and  
> have been tested extensively. If there are problems in other tools, we  
> need to look carefully at the actions that lead to the problems. I can't  
> answer off the top of my head whether there are cut and paste problems  
> in the Archetype Editor, but I don't believe there should be any  
> problems in its ADL serialisation. Perhaps there are inconsistencies in  
> the XML serialiser - it is a separate piece of code. We should certainly  
> get it fixed at source if there are any problems, but I am sure they  
> will be in the code, not the archetype files themselves.

So just to be certain, if a user has some existing textual content (e.g.  
a description of some construct) in a non-UTF-8 source (e.g. a Word  
document or HTML page) and pastes into a text area in the Archetype editor:  
A) All (mappable) non-UTF-8 chars would be transformed into UTF-8 chars?  
B) What about unmappable chars etc ?

e.g. in our WYSIWYG XHTML editor for HL7 Mif we had to put in a decoder e.g.

String charSet = "UTF-8";

\<cut\>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Charset charset = Charset.forName(charSet);  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CharsetDecoder dec = charset.newDecoder();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dec.onMalformedInput(CodingErrorAction.IGNORE);  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dec.onUnmappableCharacter(CodingErrorAction.IGNORE);

i.e. it just drops the chars which are either malformed or unmappable.

What happens wrt the Archetype editor?

Adam

---

<div class="post-metadata">

**Author:** ![Peter\_Gummer1](https://discourse.openehr.org/letter_avatar_proxy/v4/letter/p/258eb7/32.png) [@Peter\_Gummer1](https://discourse.openehr.org/u/Peter_Gummer1)\
**Post date:** [3 November 2008 21:29 UTC](https://discourse.openehr.org/t/byte-order-marks/14842/6 "2008-11-03T21:29:27Z")

</div>

Adam Flinton wrote:

> So just to be certain, if a user has some existing textual content (e.g.  
> a description of some construct) in a non-UTF-8 source (e.g. a Word  
> document or HTML page) and pastes into a text area in the Archetype editor:  
> A) All (mappable) non-UTF-8 chars would be transformed into UTF-8 chars?  
> B) What about unmappable chars etc ?
> 
> ...
> 
> What happens wrt the Archetype editor?  
> &nbsp;&nbsp;  
> Archetype Editor is built with standard .NET controls, which look after  
> those details. It's all just standard operating system stuff. A lot of  
> people from various cultures have been working with it, so this has been  
> well tested.

The only bug reported in 18 months has been in a DataGrid that we're  
using, where entering a special character by keying its code on the  
numeric keypad whilst holding down the Alt key can cause the wrong  
character to appear in the wrong row.

- Peter