Home arrow static arrow Java Programming [Archive] - reading files and subdirectories under a directory
Warning: Creating default object from empty value in /www/htdocs/w008deb8/wiki/components/com_staticxt/staticxt.php on line 51
Java Programming [Archive] - reading files and subdirectories under a directory
This topic has 6 replies on 1 page.

Posts:174
Registered: 11/30/00
reading files and subdirectories under a directory  
Jun 29, 2004 6:54 PM



 
I have to build an application that reads more than 10000 files and 500 directories in dev environment and qa environment, so altogether i have to read more than 20000 files and 1000 directories. when i implement my current code, it takes forever.

i can't figure out to make it read faster. this is my code.

// Process only files under dir
public static void visitAllFiles(File dir) {
if (dir.isDirectory()) {
String[] children = dir.list();
for (int i=0; i<children.length; i++) {
visitAllFiles(new File(dir, children));
}
} else {
process(dir);
}
}

i know this is the standard code, so i am just wondering if anybody has a better solution... i'd appreciated it..

 

Posts:183
Registered: 5/28/04
Re: reading files and subdirectories under a directory  
Jun 29, 2004 8:02 PM (reply 1 of 6)



 
What does process(dir) method do? Try commenting that out and running, it may not be a read problem at all.

This is the same code that I have used in the past, directory.list() with an isDirectory() test and recursion. If you find something that works better, please post!

-Scott
http://www.swiftradius.com
 

Posts:24,517
Registered: 98-02-27
Re: reading files and subdirectories under a directory  
Jun 29, 2004 8:40 PM (reply 2 of 6)



 
What does process(dir) method do?

Agreed this is important.

Check out the example code in this [url http://forum.java.sun.com/thread.jsp?forum=31&thread=437864&start=11&range=1]thread[/url].

I ran the above code against my JDK1.4.1 directory which has 974 directories and 14392 files.

When I run the code as is it takes 12 seconds. (I'm running on an old 400Mhz computer)

when I remove the updateTable(...) method it takes 6 seconds.

So in my case the processing doubled the time.

One minor difference in my recursion routine, is that I don't create a new File(...) when I recurse each sub directory. This won't be the bottleneck, but it will reduce the numuber of Objects to create and garbage collect.
 

Posts:174
Registered: 11/30/00
Re: reading files and subdirectories under a directory  
Jun 29, 2004 9:00 PM (reply 3 of 6)



 
the one that i just posted is not a complete method. for "process(dir)" line, it just saves file name and file path in Vector object for future use like comparing files(from another environment) and copying and deleting.

public static void visitAllFiles(File dir) {
if (dir.isDirectory()) {
String[] children = dir.list();
for (int i=0; i<children.length; i++) {
visitAllFiles(new File(dir, children));
}
} else {
process(dir);
}
}

so i am just wondering how to make my application speed up better. my application just compares two similar environments(dev, qa) and deletes files from qa if files existing in dev don't exist in qa. it just makes two environments identical. but there are so many files more than 20000 and 500 directories.

is there any better solutions?
 

Posts:6
Registered: 10/27/03
Re: reading files and subdirectories under a directory  
Jun 30, 2004 4:13 AM (reply 4 of 6)



 
Hi,

maybe you shouldn't retrieve the file names but the files directly instead:
public static void visitAllFiles(File dir) {    if (dir.isDirectory()) {        File[] children = dir.listFiles();        for (int i=0; i<children.length; i++) {            visitAllFiles(children[i]);        }    } else {        process(dir);    }}

Have you tried with "process(dir)" commented out?
 

Posts:49
Registered: 2/5/04
Re: reading files and subdirectories under a directory  
Jun 30, 2004 6:09 AM (reply 5 of 6)



 
Why dont u use the concept of threading
Assign each directory to a new thread..
Something of this kind....

if(f.isDirectory())	{            File tmp [] = f.listFiles();            if(tmp != null)	{	int len = tmp.length;	for(int i = 0 ; i < len ; i++)	{	      if(tmp[i].isDirectory())	{	              MyThread thread = new MyThread(urVectorObject, tmp[i]);	              thread.start();	     }	}           }}  

And in the MyThread class which extends Thread u can implement the functionality accordingly.
ie., what u r presently doing in visitAllFiles.
Just change the method signature to
visitAllFiles(urVectorObject, currentDirectory)

Hope this helps
 

Posts:24,517
Registered: 98-02-27
Re: reading files and subdirectories under a directory  
Jun 30, 2004 12:44 PM (reply 6 of 6)



 
, it just saves file name and file path in Vector object for future use

Do you use

new Vector()

This will allocate a Vector of size 16 I believe. When this is full it will create a new Vector twice the size and then copy everything. Of course this will get repeated multiple times which is not very efficient.

Try using:

new Vector(20000);
 
This topic has 6 replies on 1 page.