Home arrow static arrow Java Programming [Archive] - StreamTokenize and double precision
Warning: Creating default object from empty value in /www/htdocs/w008deb8/wiki/components/com_staticxt/staticxt.php on line 51
Java Programming [Archive] - StreamTokenize and double precision
This topic has 6 replies on 1 page.

Posts:4
Registered: 7/1/04
StreamTokenize and double precision  
Jul 1, 2004 8:54 AM



 
I am using the Streamtokenizer to read in some info from a text file.
I noticed that the following number in my text file does not get read in correctly:
The input: 168204985223372809
The output:
nval 1.682049852233728E17
ToLong 4864640403720995584
Raw 4864640403720995584
Cast 168204985223372800
longVal 168204985223372800
StringToken[n=1.682049852233728E17], line 1

I am especially interested in the longValue number. The number is still way off from Long.MAX.

It even gets rounded incorrectly. Any pointers, code snippet below:

FileReader in = new FileReader("c:/temp/OvObsNotification.txt");
StreamTokenizer st = new StreamTokenizer(new BufferedReader(in));
st.commentChar('#');
st.slashSlashComments(true);
st.slashStarComments(true);
st.eolIsSignificant(true);

int word = 0;

boolean running = true;
while (running) {
st.nextToken();
switch (st.ttype) {
case StreamTokenizer.TT_EOF: // last line therefor
running = false; // must fall tru
case StreamTokenizer.TT_EOL:
word = 0;
break;

case StreamTokenizer.TT_NUMBER:
if (word == 0) {

double entityID = st.nval;
System.out.println("nval " + st.nval);
System.out.println("ToLong " + Double.doubleToLongBits(st.nval));
System.out.println("Raw " + Double.doubleToRawLongBits(st.nval));
System.out.println("Cast " + (long)st.nval);
Double d = new Double(st.nval);
System.out.println("longVal" + d.longValue());
System.out.println("String" + st.toString());
}

word++;
break;

default:
break;
} // end-switch
} // end-while

in.close();
}
 

Posts:2
Registered: 6/11/04
Re: StreamTokenize and double precision  
Jul 2, 2004 5:59 AM (reply 1 of 6)



 
Your problem is with double precision numbers. Although they can hold numbers that are very high, they are limited in their accuracy. Your large number is stored as a double in the StreamTokeniser, so 168204985223372809 is stored as 1.682049852233728E17, losing the last two digits of accuracy. Subsequent conversions to long cannot re-create the lost digits. Unfortunately, all numbers in a StreamTokeniser are handled as double's, with no alternative for nval. Your options are a StreamTokeniser which does not call parseNumbers, or not use StreamTokeniser at all.

Unfortunately for the first idea, all StreamTokenisers call parseNumbers in a private CTOR you can't override, so you need to reverse its effects using wordChars. Then, you will get all digits as sval's, which you will have to convert to long manually.

Good luck!
 

Posts:1,183
Registered: 1/23/02
Re: StreamTokenize and double precision  
Jul 2, 2004 7:03 AM (reply 2 of 6)



 
[url http://docs.sun.com/source/806-3568/ncg_goldberg.html]What Every Computer Scientist Should Know About Floating Point Arithmetic[/url]

Types double and long are both 64 bits wide, but double uses some of the bits for the exponent.
 

Posts:4
Registered: 7/1/04
Re: StreamTokenize and double precision  
Jul 2, 2004 8:41 AM (reply 3 of 6)



 
[url
http://docs.sun.com/source/806-3568/ncg_goldberg.html]W
at Every Computer Scientist Should Know About Floating
Point Arithmetic[/url]

Types double and long are both 64 bits wide, but
double uses some of the bits for the exponent.

No need to get snotty.
Anyways, I was hoping by not calling parseNumbers it would return it as a string value.
I was also expecting that if I did call parseNumbers, it would recognize 1.682049852233728E17.

I'll write my own.
 

Posts:1,183
Registered: 1/23/02
Re: StreamTokenize and double precision  
Jul 2, 2004 9:02 AM (reply 4 of 6)



 
No need to get snotty.

The title of the article is the author's (David Goldberg's).

Anyways, I was hoping by not calling parseNumbers it would return it as a string value.

Yes, it's rather annoying that StreamTokenizer enables parseNumbers by default. You can clear that property by calling resetSyntax just after you create the tokenizer.

I was also expecting that if I did call parseNumbers, it would recognize 1.682049852233728E17.

What do you mean by "recognize"?
 

Posts:4
Registered: 7/1/04
Re: StreamTokenize and double precision  
Jul 2, 2004 10:37 AM (reply 5 of 6)



 
No need to get snotty.

The title of the article is the author's (David
Goldberg's).

Ah, missed that you just included a link....

Anyways, I was hoping by not calling parseNumbers it
would return it as a string value.

Yes, it's rather annoying that StreamTokenizer enables
parseNumbers by default. You can clear that property
by calling resetSyntax just after you create the
tokenizer.

Thanks, I'll try that.


I was also expecting that if I did call
parseNumbers, it would recognize 1.682049852233728E17.

What do you mean by "recognize"?

That StreamTokenizer would recognize this type of notation as a number.
If I have the following line in my input file: 1.682049852233728E17
Streamtokenizer does the following:
first time around: nval = 1.682049852233728
second time : sval=E17
 

Posts:4
Registered: 7/1/04
Re: StreamTokenize and double precision  
Jul 6, 2004 8:27 AM (reply 6 of 6)



 
In the end I just rewrote what I had and replaced streamtokenizer with my own stuff that actually works as expected. thanks for the suggestions.
 
This topic has 6 replies on 1 page.