Home arrow static arrow Java Programming [Archive] - java.lang.OutOfMemoryError - Big problem!! Urgent
Warning: Creating default object from empty value in /www/htdocs/w008deb8/wiki/components/com_staticxt/staticxt.php on line 51
Java Programming [Archive] - java.lang.OutOfMemoryError - Big problem!! Urgent
3 Duke Stars available
This topic has 31 replies on 3 pages.    « Previous | 1 | 2 | 3 | Next »

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:40 AM (reply 15 of 31)



 
Yeah. That code keeps putting things on the stack but never permanetly removes anything from it.

What it's doing is creating a huge in-memory model of the XML document. That's what DOM does.

What you want to do is as the elements come in, write to the file as soon as you have enough info to write anything out. Once you have finished with that element or those elements, you need to dereference them.

There's something weird about the way the stack is used in that code. It's popping, adding things to the parent and then pushing the top element back onto the stack.

Generally, you should be pushing a new element in startElement method, and popping that element on the endElement method. If you need to hold onto those objects past the endElement method, you should put it into another data structure. Don't kluge up the stack.
 

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:43 AM (reply 16 of 31)



 
I must read xml,, its impossible generate txt file,
because xml is my data source

I understand. But step away from the code for a second and lets think this out. You need a plan.

If you were going to write the txt file manually, do you need to have the entire XML file in front of you to do so? If not, good. If you do, what info do you need that you can only get from the entire file?
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:52 AM (reply 17 of 31)



 
I need every info from entire file. This xml comes from Brazilian Federal Reserve, every info is important to generate a txt file
 

Posts:19,725
Registered: 9/26/01
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:53 AM (reply 18 of 31)



 
It's time for another famous reply:
Jeepers!
 

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:58 AM (reply 19 of 31)



 
I need every info from entire file. This xml comes
from Brazilian Federal Reserve, every info is
important to generate a txt file

So you can't write a single line out until you have the entire XML file available? Why not?
 

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 8:04 AM (reply 20 of 31)



 
I need every info from entire file. This xml comes
from Brazilian Federal Reserve, every info is
important to generate a txt file

We are probably dealing with a little language barrier here. I'm not saying not to read in the entire file, I'm asking whether you have to read the entire file in before you can start writing the text file. For example:

let's say I have this XML file.

<colors>
<red warm="true"/>
<yellow warm="true"/>
<green warm="false"/>
</colors>

And I want to write it to this format:

[colors][red:warm][yellow:warm][green:cool]


Do I need to have the entire XML tree in memory to start writing the txt file? No. As I reach each node, I can write it's data to the file. I don't need to know that yellw is warm in order to write out that red is warm.
 

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 8:07 AM (reply 21 of 31)



 
Also, have you looked into XSLT?
 

Posts:3,258
Registered: 00-08-28
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 8:17 AM (reply 22 of 31)



 
if you ask me XSLT would be the right way for such a project.
 

Posts:3,258
Registered: 00-08-28
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 8:19 AM (reply 23 of 31)



 
[url http://www-106.ibm.com/developerworks/xml/library/x-xslt/] here is a brief intro[/url]
 

Posts:352
Registered: 7/18/03
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 8:51 AM (reply 24 of 31)



 

I think that Rational Purify is smart enough now to tell you about Java memory usage. You can download a a 15-day trial version at

http://www14.software.ibm.com/webapp/download/downloadaz.jsp

Here is a short article they have about using it for Java:

http://www-106.ibm.com/developerworks/rational/library/1791.html

If this helps you find your memory leak, then you have to decide how/when to remove all your references to any objects you are retaining and change your code to do so.

Obviously, for every call to Stack.push(), you should eventually call Stack.pop() .
 

Posts:352
Registered: 7/18/03
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 12:51 PM (reply 25 of 31)



 

You might also want to try JProfiler to find your leak

http://www.ej-technologies.com/products/jprofiler/overview.html
 

Posts:37,103
Registered: 3/30/99
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 1:06 PM (reply 26 of 31)



 
If you want a little bit cleaner code for processing node by node, you could try the struts digester from jakarta.apache.org. (Actually, it may have moved out of struts and into its own project.) It's built on top of SAX, I think, but it handles some of the lower level plumbing for you and lets you specify declaratively the XPath (or XPath-like) sub-structures you want to process and what to do when you hit them. It's fairly elegant, IMO.

There is a bit of a learning curve--not much of one, but since you're about to fall off the cliff, I don't suppose you'll be interested.

A bit of unsolicited advice: I suspect that, given that you're dealing with money here, it's going to be more important for this to be correct and robust than for it to be delivered by whatever your urgent deadline is. Given what you've posted so far, I doubt you'll be able to have it now and correct and robust. I suggest you bite the bullet and inform your boss that you need more time.

As for not being able to give the VM more memory, I don't know your particular situation, but a couple hundred MB is not all that much, especially on a production box in a financial institution. I would think if this were that important, your employer would be able to scrounge up the dough for 1GB or so. The cost of the memory is probably less than your pay for the time you're spending trying to fix this.

(Note that I think it's still better to read the file piecemeal, rather than building the whole DOM. Just pointing out options.)
 

Posts:31,095
Registered: 4/30/99
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 1:43 PM (reply 27 of 31)



 
if you ask me XSLT would be the right way for such a
project.

I would say so too except for the size of the input document. XSLT transformers build an in-memory tree representation of the input document, and a rule of thumb I have seen in that context is to provide 10 times as much memory as the document's size. That would be 1.37 GB for the OP's example with no guarantee that there would never be larger files to process.

I would question the decision to produce XML files of that size, too, but I am sure the OP has no control over that decision. So it looks to me like a solution based on SAX would be the way to go.

PC²
 

Posts:352
Registered: 7/18/03
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 2:55 PM (reply 28 of 31)



 

One other thread that mentions a few other tools:

http://forum.java.sun.com/thread.jsp?thread=505602&forum=31

and also there is an Eclipse profiler

http://sourceforge.net/projects/eclipsecolorer/
 

Posts:352
Registered: 7/18/03
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 3:03 PM (reply 29 of 31)



 

There is one other way that is built-in to Java but very ugly.

Just run with

java -Xrunhprof:heap=dump

and look in the HUGE (many megabytes) text file

java.hprof.txt

This will tell you what it leaking, but not necessarily why.

If you try this, try to use just a small part of your XML file and you should call System.gc() before you exit.
 
This topic has 31 replies on 3 pages.    « Previous | 1 | 2 | 3 | Next »