Home arrow static arrow Java Programming [Archive] - java.lang.OutOfMemoryError - Big problem!! Urgent
Warning: Creating default object from empty value in /www/htdocs/w008deb8/wiki/components/com_staticxt/staticxt.php on line 51
Java Programming [Archive] - java.lang.OutOfMemoryError - Big problem!! Urgent
3 Duke Stars available
This topic has 31 replies on 3 pages.    1 | 2 | 3 | Next »

Posts:14
Registered: 6/3/04
java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:19 AM



 
I have a big xml file (137 MB), Someone else has recommended me use SAX parser, to solve Memory problem, but i hadnt success. The problem is still happening..OutOfMemoryError.
I need read xml file and create a new .txt file from xml.....Is there another API to parser this very big XML file without problem ????

Thanks a lot

 

Posts:5,119
Registered: 16.08.01
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:28 AM (reply 1 of 31)



 
Can't you allocate more memory for the JVM using the -Xmx flag on startup?
 

Posts:21,719
Registered: 98-02-20
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:28 AM (reply 2 of 31)



 
A SAX parser is a good idea, because you don't create a DOM tree in memory and make the problem worse.

But I think the idea is that you should override SAX event models to get the items you need without having to hold the entire XML file in memory at once. You need to process it on the fly.

Can't tell what you really need to do, but I think that's the idea behind the SAX parser recommendation. You've got to do things in such a way that you don't hold the XML file in memory at the same time. Just cherry pick the parts you need as you go.

%
 

Posts:24,036
Registered: 2/3/03
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:28 AM (reply 3 of 31)



 
Don't flag your question as urgent, even if it is for you. Claiming urgency is very likely to be counter-productive: most forum regulars will simply ignore such messages as rude and selfish attempts to elicit immediate and special attention.

Besides, unless the entire Bolivian army is about to chase you over the edge of a cliff, your problem probably isn't as urgent as you think. :o)

Please be a little more explicit about what you're doing, what you expect to happen, and what you actually observe.

Please post a short, concise, executable example of what you're trying to do. This does not have to be the actual code you are using. Write a small example that demonstrates your intent, and only that. Wrap the code in a class and give it a main method that runs it - if we can just copy and paste the code into a text file, compile it and run it without any changes, then we can be sure that we haven't made incorrect assumptions about how you are using it.

Post your code between [code] and [/code] tags as described in [url=http://forum.java.sun.com/features.jsp#Formatting]Formatting Help[/url] on the message entry page. Cut and paste the code, rather than re-typing it (re-typing often introduces subtle errors that make your problem difficult to troubleshoot). Please preview your post when posting code.

Please assume that we only have the core API. We have no idea what SomeCustomClass is, and neither does our collective compiler.

If you have an error message, post the exact, complete error along with a full stack trace, if possible. Make sure you're not swallowing any Exceptions.
 

Posts:5
Registered: 6/16/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:30 AM (reply 4 of 31)



 
Have you tried Dom/JDom?

Hope that this helps!
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:35 AM (reply 5 of 31)



 
I have a really urgent big problem,,I work in a big bank,,and im trying to solve a big URGENT problem. Sorry if i didnt send clear message, but my problem is urgent.
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:37 AM (reply 6 of 31)



 
I cant allocate more memory, because there are another proccess running
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:38 AM (reply 7 of 31)



 
I didnt try DOM/JDOM because i received the message below.

"A SAX parser is a good idea, because you don't create a DOM tree in memory and make the problem worse"
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:43 AM (reply 8 of 31)



 
My java code is based java code below. This java code sample is available on http://www.onjava.com/pub/a/onjava/2002/06/26/xml.html?page=1

File file = new File( args[1] );
InputSource src = new InputSource( new FileInputStream( file ) );

if( args[0].equals( "SAX" ) ) {
System.out.println( "--- SAX ---" );

SaxCatalogUnmarshaller saxUms = new SaxCatalogUnmarshaller();

XMLReader rdr = XMLReaderFactory.
createXMLReader( "org.apache.xerces.parsers.SAXParser" );
rdr.setContentHandler( saxUms );
rdr.parse( src );

catalog = saxUms.getCatalog();


class Catalog {
private Vector books;
private Vector magazines;

public Catalog() {
books = new Vector();
magazines = new Vector();
}

public void addBook( Book rhs ) {
books.addElement( rhs );
}
public void addMagazine( Magazine rhs ) {
magazines.addElement( rhs );
}

public String toString() {
String newline = System.getProperty( "line.separator" );
StringBuffer buf = new StringBuffer();

buf.append( "--- Books ---" ).append( newline );
for( int i=0; i<books.size(); i++ ){
buf.append( books.elementAt(i) ).append( newline );
}

buf.append( "--- Magazines ---" ).append( newline );
for( int i=0; i<magazines.size(); i++ ){
buf.append( magazines.elementAt(i) ).append( newline );
}

return buf.toString();
}
}

class SaxCatalogUnmarshaller extends DefaultHandler {
private Catalog catalog;

private Stack stack;
private boolean isStackReadyForText;

private Locator locator;

//


public SaxCatalogUnmarshaller() {
stack = new Stack();
isStackReadyForText = false;
}

public Catalog getCatalog() { return catalog; }

//


callbacks:

public void setDocumentLocator( Locator rhs ) { locator = rhs; }

//

public void startElement( String uri, String localName, String qName,
Attributes attribs ) {

isStackReadyForText = false;

// if next element is complex, push a new instance on the stack
// if element has attributes, set them in the new instance
if( localName.equals( "catalog" ) ) {
stack.push( new Catalog() );

}else if( localName.equals( "book" ) ) {
stack.push( new Book() );

}else if( localName.equals( "magazine" ) ) {
stack.push( new Magazine() );

}else if( localName.equals( "article" ) ) {
stack.push( new Article() );
String tmp = resolveAttrib( uri, "page", attribs, "unknown" );
((Article)stack.peek()).setPage( tmp );
}
// if next element is simple, push StringBuffer
// this makes the stack ready to accept character text
else if( localName.equals( "title" ) || localName.equals( "author" ) ||
localName.equals( "name" ) || localName.equals( "headline" ) ) {
stack.push( new StringBuffer() );
isStackReadyForText = true;
}
// if none of the above, it is an unexpected element
else{
// do nothing
}
}

//


public void endElement( String uri, String localName, String qName ) {

// recognized text is always content of an element
// when the element closes, no more text should be expected
isStackReadyForText = false;

// pop stack and add to 'parent' element, which is next on the stack
// important to pop stack first, then peek at top element!
Object tmp = stack.pop();

if( localName.equals( "catalog" ) ) {
catalog = (Catalog)tmp;

}else if( localName.equals( "book" ) ) {
((Catalog)stack.peek()).addBook( (Book)tmp );

}else if( localName.equals( "magazine" ) ) {
((Catalog)stack.peek()).addMagazine( (Magazine)tmp );

}else if( localName.equals( "article" ) ) {
((Magazine)stack.peek()).addArticle( (Article)tmp );
}
// for simple elements, pop StringBuffer and convert to String
else if( localName.equals( "title" ) ) {
((Book)stack.peek()).setTitle( tmp.toString() );

}else if( localName.equals( "author" ) ) {
((Book)stack.peek()).setAuthor( tmp.toString() );

}else if( localName.equals( "name" ) ) {
((Magazine)stack.peek()).setName( tmp.toString() );

}else if( localName.equals( "headline" ) ) {
((Article)stack.peek()).setHeadline( tmp.toString() );
}
// if none of the above, it is an unexpected element:
// necessary to push popped element back!
else{
stack.push( tmp );
}
}

//



public void characters( char[] data, int start, int length ) {

// if stack is not ready, data is not content of recognized element
if( isStackReadyForText == true ) {
((StringBuffer)stack.peek()).append( data, start, length );
}else{
// read data which is not part of recognized element
}
}

//

private String resolveAttrib( String uri, String localName,
Attributes attribs, String defaultValue ) {

String tmp = attribs.getValue( uri, localName );
return (tmp!=null)?(tmp):(defaultValue);
}
}
 

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 6:54 AM (reply 9 of 31)



 
I'm trying to remember why I think this will help. change:

InputSource src = new InputSource( new FileInputStream( file ) );

to this,

InputSource src = new InputSource( new BufferedReader( FileInputStream( file ) ) );

and let me know if that makes any difference.
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:07 AM (reply 10 of 31)



 
To each xml tag im create Java Object, Can be a problem ??????

if( qName.equalsIgnoreCase( "doc3082" ) ) {
// System.out.println("XPTO") ;
stack.push( new ArquivoXMLBean() );

String tmp = resolveAttrib( uri, "DtBase", attribs, "unknown" );
((ArquivoXMLBean)stack.peek()).setDtbase(tmp);

tmp = resolveAttrib( uri, "CNPJ", attribs, "unknown" );
((ArquivoXMLBean)stack.peek()).setCnpj(tmp);

tmp = resolveAttrib( uri, "Protocolo", attribs, "unknown" );
((ArquivoXMLBean)stack.peek()).setProtocolo(tmp);

}else if( qName.equalsIgnoreCase( "clicons" ) ) {
stack.push( new Clicons() );
// System.out.println("XPTO") ;

}else if( qName.equalsIgnoreCase( "opcli" ) ) {
stack.push( new Opcli() );
// System.out.println("XPTO") ;

}else if( qName.equalsIgnoreCase( "virvenc" ) ) {
stack.push( new Virvenc() );
String tmp = resolveAttrib( uri, "CodVenc", attribs, "unknown" );
((Virvenc)stack.peek()).setCodvenc( tmp );
}

XML Structure below

<?xml version="1.0" encoding="ISO-8859-1" ?>
- <Doc3082 DtBase="2004-03" CNPJ="60701190" Protocolo="S0000103">
<DtGeracao>2004-05-20T13:46:00</DtGeracao>
<PercDocProcess>82.16</PercDocProcess>
<VolPercProcess>3525.02</VolPercProcess>
- <CliCons>
<Cli TpCli="2">00289027</Cli>
</CliCons>
- <CliCons>
<Cli TpCli="2">04584810</Cli>
</CliCons>
- <CliCons>
<Cli TpCli="2">00000028</Cli>
<QtdOp>25</QtdOp>
<QtdIf>7</QtdIf>
- <OpCli>
<Mod>0101</Mod>
<VlrVenc CodVenc="110">1.05</VlrVenc>
</OpCli>
- <OpCli>
<Mod>0201</Mod>
<VlrVenc CodVenc="110">34789.55</VlrVenc>
<VlrVenc CodVenc="120">21691.03</VlrVenc>
</OpCli>
- <OpCli>
<Mod>0205</Mod>
<VlrVenc CodVenc="110">5732.44</VlrVenc>
<VlrVenc CodVenc="120">5124.37</VlrVenc>
<VlrVenc CodVenc="130">5175.09</VlrVenc>
<VlrVenc CodVenc="140">16033.58</VlrVenc>
<VlrVenc CodVenc="150">5529.31</VlrVenc>
</OpCli>
- <OpCli>
<Mod>0299</Mod>
<VlrVenc CodVenc="140">45182.59</VlrVenc>
</OpCli>
- <OpCli>
<Mod>0299</Mod>
<VincME>S</VincME>
<VlrVenc CodVenc="120">52991.09</VlrVenc>
</OpCli>
- <OpCli>
<Mod>0401</Mod>
<VlrVenc CodVenc="110">863.97</VlrVenc>
<VlrVenc CodVenc="120">841.77</VlrVenc>
<VlrVenc CodVenc="130">819.43</VlrVenc>
<VlrVenc CodVenc="140">2332.24</VlrVenc>
<VlrVenc CodVenc="150">4144.62</VlrVenc>
<VlrVenc CodVenc="160">2597.68</VlrVenc>
</OpCli>
- <OpCli>
<Mod>0402</Mod>
<VlrVenc CodVenc="110">1935.62</VlrVenc>
<VlrVenc CodVenc="120">1884.71</VlrVenc>
<VlrVenc CodVenc="130">1833.52</VlrVenc>
<VlrVenc CodVenc="140">4481.17</VlrVenc>
<VlrVenc CodVenc="150">528.02</VlrVenc>
</OpCli>
- <OpCli>
<Mod>1201</Mod>
<VlrVenc CodVenc="110">1986.61</VlrVenc>
<VlrVenc CodVenc="120">1933.94</VlrVenc>
<VlrVenc CodVenc="130">1882.65</VlrVenc>
<VlrVenc CodVenc="140">5353.68</VlrVenc>
<VlrVenc CodVenc="150">9495.40</VlrVenc>
<VlrVenc CodVenc="160">14959.18</VlrVenc>
<VlrVenc CodVenc="165">1042.30</VlrVenc>
</OpCli>

 

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:15 AM (reply 11 of 31)



 
To each xml tag im create Java Object, Can be a
problem ??????

Depends on what you do with them. I don't care for that code you posted. What is it you need to do with the data you read in from the XML file?
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:21 AM (reply 12 of 31)



 
I must read xml and , process this tag using my business rules, and generate a new .txt file. This file will be send to mainframe.
 

Posts:13,769
Registered: 00-11-29
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:29 AM (reply 13 of 31)



 
OK. Creating a new Object for each element shouldn't be a problem as long as you don't hold onto it forever. I'm not positive but it looks like that's what this code is doing. Can you write the text file out a little at a time without looking at the entire document?
 

Posts:14
Registered: 6/3/04
Re: java.lang.OutOfMemoryError - Big problem!! Urgent  
Jun 16, 2004 7:39 AM (reply 14 of 31)



 
InputSource src = new InputSource( new BufferedReader( FileReader( file ) ) );
and let me know if that makes any difference.

This java code below helped me in 15%

Made difference, but just 15% at cpu and memory, but the problem is still happening.

I must read xml,, its impossible generate txt file, because xml is my data source
 
This topic has 31 replies on 3 pages.    1 | 2 | 3 | Next »