# Unhandled exception parsing ADL

**Category:** [Reference Implementation: Java (archive)](https://discourse.openehr.org/c/reference-implementation-java-archive/154)
**Created:** 2008-10-23 08:28 UTC
**Views:** 3
**Replies:** 4
**URL:** https://discourse.openehr.org/t/unhandled-exception-parsing-adl/16106

---

## Post #1 by @Adam_Flinton

I&#39;ve built an ant task for generating XML from ADL &amp; running it against
out adl archetypes dies when an unhandled error pops up\.

Exception in thread &quot;main&quot; se\.acode\.openehr\.parser\.TokenMgrError:
Lexical error at line 1, column 1\.  Encountered: &quot;\\u00ef&quot; \(239\), after : &quot;&quot;
&nbsp;&nbsp;&nbsp;&nbsp;at
se\.acode\.openehr\.parser\.ADLParserTokenManager\.getNextToken\(ADLParserTokenManager\.java:27554\)
&nbsp;&nbsp;&nbsp;&nbsp;at
se\.acode\.openehr\.parser\.ADLParser\.jj\_consume\_token\(ADLParser\.java:7061\)
&nbsp;&nbsp;&nbsp;&nbsp;at se\.acode\.openehr\.parser\.ADLParser\.archetype\(ADLParser\.java:214\)
&nbsp;&nbsp;&nbsp;&nbsp;at se\.acode\.openehr\.parser\.ADLParser\.parse\(ADLParser\.java:101\)

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;final public Archetype archetype\(\) throws ParseException, Exception \{
&lt;cut&gt;
&nbsp;&nbsp;&nbsp;&nbsp;jj\_consume\_token\(SYM\_ARCHETYPE\);
&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;final private Token jj\_consume\_token\(int kind\) throws ParseException \{
&nbsp;&nbsp;&nbsp;&nbsp;Token oldToken;
&nbsp;&nbsp;&nbsp;&nbsp;if \(\(oldToken = token\)\.next \!= null\) token = token\.next;
&nbsp;&nbsp;&nbsp;&nbsp;else token = token\.next = token\_source\.getNextToken\(\);
&nbsp;&nbsp;&nbsp;
&lt;cut&gt;

public Token getNextToken\(\)

&lt;cut&gt;
&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;throw new TokenMgrError\(EOFSeen, curLexState, error\_line,
error\_column, error\_after, curChar, TokenMgrError\.LEXICAL\_ERROR\);

I\.e\. the Token jj\_consume\_token\(\) method only throws a parse exception
where getNextToken\(\) :

A\) Throws a TokenMgrError within the body of the method
B\) Doesn&#39;t then throw it at the method level \(e\.g\. public Token
getNextToken\(\) throws TokenMgrError \{\} \)
C\) Token jj\_consume\_token\(\) doesn&#39;t then either catch or throw the
TokenMgrError\.

Adam

---

## Post #2 by @Adam_Flinton

The problem resolves to the \(optional\) Byte Order Mark for UTF\-8\.
Various times people have opened ADL files with text editors &amp; some
introduce the BOM upon saving\.

I have written a little class which strips the BOM but given the parser
looks to be generated I am unsure how to integrate it\.

The byte array to check for/starts with is:

&nbsp;&nbsp;&nbsp;&nbsp;private static final byte UTF8\_BOM\_1 = \(byte\) 0xef;
&nbsp;&nbsp;&nbsp;&nbsp;private static final byte UTF8\_BOM\_2 = \(byte\) 0xbb;
&nbsp;&nbsp;&nbsp;&nbsp;private static final byte UTF8\_BOM\_3 = \(byte\) 0xbf;

\.\.\.

final byte\[\] utf8Bom = \{UTF8\_BOM\_1, UTF8\_BOM\_2, UTF8\_BOM\_3\};

And then this needs to be removed prior to parsing

Any takers?

Any idea at which level this would be best integrated so that I can do
it &amp; then send the changed files to this list or indeed how do I get
commit rights etc?

Adam

Adam Flinton wrote:

---

## Post #3 by @system

Hi Adam,

there is a parser rule in the adl\.jj that ignores the BOM \(ll\. 224\):

&lt; \* &gt; SKIP : /\* WHITE SPACE \*/
\{
&nbsp;&nbsp;&quot; &quot;
> &quot;\\t&quot;
> &quot;\\n&quot;
> &quot;\\r&quot;
> &quot;\\f&quot;
> &quot;\\ufeff&quot; /\* UTF\-8 Byte Order Mark \*/

\}

Not sure why this doesn&#39;t work for you?

Cheers
Sebastian

Adam Flinton wrote:

---

## Post #4 by @Adam_Flinton

Sebastian Garde wrote:
> Hi Adam,
>
> there is a parser rule in the adl\.jj that ignores the BOM \(ll\. 224\):
>
> &lt; \* &gt; SKIP : /\* WHITE SPACE \*/
> \{
>  &quot; &quot;
> &gt; &quot;\\t&quot;
> &gt; &quot;\\n&quot;
> &gt; &quot;\\r&quot;
> &gt; &quot;\\f&quot;
> &gt; &quot;\\ufeff&quot; /\* UTF\-8 Byte Order Mark \*/
> \}
>
> Not sure why this doesn&#39;t work for you?
>
Good question\.\.\.\.\.\. However it doesn&#39;t seem to\.\.\.\.Have you tried firing
in ADL which have a BOM?

Also\.\.\. are you sure &quot;\\ufeff&quot; is correct?

Not:

&quot;\\u00ef&quot; \(i\.e\. decimal 239\)\.

Also the BOM is:

> 239 187 191  i\.e\. ||EF BB BF &amp; not just 239|

I&#39;ve integrated the fix into my own ant task though :

&nbsp;&nbsp;&nbsp;&nbsp;// For BOM removal
&nbsp;&nbsp;&nbsp;&nbsp;private static final byte UTF8\_BOM\_1 = \(byte\) 0xef;
&nbsp;&nbsp;&nbsp;&nbsp;private static final byte UTF8\_BOM\_2 = \(byte\) 0xbb;
&nbsp;&nbsp;&nbsp;&nbsp;private static final byte UTF8\_BOM\_3 = \(byte\) 0xbf;
&nbsp;&nbsp;&nbsp;&nbsp;final byte\[\] utf8Bom = \{ UTF8\_BOM\_1, UTF8\_BOM\_2, UTF8\_BOM\_3 \};
&nbsp;&nbsp;&nbsp;&nbsp;byte\[\] bomBuffer = new byte\[utf8Bom\.length\];
&nbsp;&nbsp;&nbsp;&nbsp;long skipL = utf8Bom\.length;

&lt;cut&gt;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;InputStream test = getInputStream\(inputFile\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;InputStream in = getInputStream\(inputFile\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OutputStream out = getOutputStream\(outputFile\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ADL2XMLConvertor conv = new ADL2XMLConvertor\(\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//System\.out\.println\(&quot;PCDTO convert\(\) OK to here
inputFile = &quot;\+inputFile\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;boolean skip = checkRemoveBOM\(test\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System\.out\.println\(&quot;skip = &quot;\+skip\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if\(skip\)\{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System\.out\.println\(&quot;skipL = &quot;\+skipL\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in\.skip\(skipL\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\}
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;err = conv\.convert\(in, out\);

&lt;cut&gt;

&nbsp;&nbsp;&nbsp;&nbsp;public boolean checkRemoveBOM\(InputStream in\) \{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;boolean containsBom = false;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try \{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;int nRead = in\.read\(bomBuffer, 0, bomBuffer\.length\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if \(nRead \!= \-1\) \{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if \(java\.util\.Arrays\.equals\(bomBuffer, utf8Bom\)\) \{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// System\.out\.write\(bomBuffer, 0, nRead\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System\.out\.println\(&quot;Contains BOM&quot;\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;containsBom = true;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\}
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\}
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\} catch \(java\.io\.IOException e\) \{
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;System\.err\.println\(&quot;I/O error occurred: &quot; \+ e\.getMessage\(\)\);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\}
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return containsBom;
&nbsp;&nbsp;&nbsp;&nbsp;\}

&amp; this works so I can get on with the generation to XML which is the
main thing for me\.

I have an XML diff task &amp; the Ocean ADL2XMl converter so I am about to
run some test runs &amp; then eventually run against the NHS ADL which
is in:

http://svn.openehr.org/knowledge/archetypes/dev-uk-nhs/adl

So I may be coming back with a list of diffs\.

TIA

Adam

---

## Post #5 by @system

Hi Adam,

This should be an easy fix. Just include your BOM1,2,3 as part of the SKIP block. Can you forward some archetype that has these BOMs for testcase?

Cheers,
Rong

---


**Canonical:** https://discourse.openehr.org/t/unhandled-exception-parsing-adl/16106
**Original content:** https://discourse.openehr.org/t/unhandled-exception-parsing-adl/16106