Home arrow static arrow Java Programming [Archive] - Splitting really huge XML files(50 GB avg size)
Warning: Creating default object from empty value in /www/htdocs/w008deb8/wiki/components/com_staticxt/staticxt.php on line 51
Java Programming [Archive] - Splitting really huge XML files(50 GB avg size)
This topic has 31 replies on 3 pages.    « Previous | 1 | 2 | 3 | Next »

Posts:197
Registered: 02.09.97
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 11:49 AM (reply 15 of 31)



 
I'm just curious, could you tell the type of Filesystem that handles single files with the size of 50GB+?????

 

Posts:2,821
Registered: 7/10/01
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 12:07 PM (reply 16 of 31)



 
I'm just curious, could you tell the type of
Filesystem that handles single files with the size of
50GB+?????


NTFS can handle files as large as 2 Terrabytes,

http://www.ntfs.com/ntfs_vs_fat.htm
 

Posts:13,769
Registered: 00-11-29
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 12:43 PM (reply 17 of 31)



 
NTFS can handle files as large as 2 Terrabytes,

Sweet. We can look forward to posts like "How to handle big (1 TB) XML file?"
 

Posts:27,518
Registered: 11/3/97
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 1:19 PM (reply 18 of 31)



 
I'm just curious, could you tell the type of
Filesystem that handles single files with the size
of
50GB+?????


NTFS can handle files as large as 2 Terrabytes,

http://www.ntfs.com/ntfs_vs_fat.htm

Solaris 8 and perhaps 7 can handle large files (64 bit range I believe.)

I suspect most up to date unix systems can handle larger file sizes.

Of course on any of the OSes it should be possible to run into applications that won't handle the larger sizes.
 

Posts:349
Registered: 1/8/04
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 2:24 PM (reply 19 of 31)



 
I suspect most up to date unix systems can handle
larger file sizes.

Macs, too:
"For Mac OS X 10.3 Panther, the maximum volume size is 16 TB (terabytes). For Mac OS X 10.2 and Mac OS X 10.2 Server, the maximum volume size is 8 TB. For earlier versions of Mac OS X, the maximum is 2 TB.

The theoretical maximum file size for a Mac OS Extended file system is millions of terabytes. In practice, the maximum file size is equivalent to the maximum volume size, except for a small amount of disk space reserved for file system information."

Makes me wonder, what systems don't support 50GB files?
 

Posts:13,769
Registered: 00-11-29
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 2:31 PM (reply 20 of 31)



 
Makes me wonder, what systems don't support 50GB
files?

Commodore 64.
 

Posts:349
Registered: 1/8/04
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 2:45 PM (reply 21 of 31)



 
Commodore 64.

What's the '64' mean, then?
 

Posts:13,769
Registered: 00-11-29
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 3:00 PM (reply 22 of 31)



 
Commodore 64.

What's the '64' mean, then?

Didn't it mean 64KB?
 

Posts:2,821
Registered: 7/10/01
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 3:01 PM (reply 23 of 31)



 
Makes me wonder, what systems don't support 50GB files?

Fat16 & Fat32 for starters.
 

Posts:13,769
Registered: 00-11-29
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 3:03 PM (reply 24 of 31)



 
What's the '64' mean, then?

Yeah, 64 K of memory.
 

Posts:349
Registered: 1/8/04
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 3:45 PM (reply 25 of 31)



 
Not 64GB, then, huh? Oh well.

I think I've still got a TI-94 around somewhere. Those cartridges probably didn't hold 50 gigs either. Maybe there's a JRE for it. That would be a hoot.
 

Posts:9,091
Registered: 8/15/01
Re: Splitting really huge XML files(50 GB avg size)  
Jul 14, 2004 4:46 PM (reply 26 of 31)



 
Not 64GB, then, huh? Oh well.

I think I've still got a TI-94 around somewhere. Those
cartridges probably didn't hold 50 gigs either. Maybe
there's a JRE for it. That would be a hoot.

I've seen a JRE for the C64 I'm sure.

The porn DTD, you need to wait for "Big Steve" at the bike shed behind the W3C office block.
 

Posts:27,518
Registered: 11/3/97
Re: Splitting really huge XML files(50 GB avg size)  
Jul 15, 2004 11:36 AM (reply 27 of 31)



 
I suspect most up to date unix systems can handle
larger file sizes.

Macs, too:
"For Mac OS X 10.3 Panther, the maximum volume size is
16 TB (terabytes). For Mac OS X 10.2 and Mac OS X 10.2
Server, the maximum volume size is 8 TB. For earlier
versions of Mac OS X, the maximum is 2 TB.

The theoretical maximum file size for a Mac OS
Extended file system is millions of terabytes. In
practice, the maximum file size is equivalent to the
maximum volume size, except for a small amount of disk
space reserved for file system information."

Makes me wonder, what systems don't support 50GB
files?

Every early system based on 32 bits.

The standard C API file io routines do not support file sizes larger than 32 signed bits (2 gigs.) So for example to handle larger file sizes on windows you have to use the non-standard C API io methods.
 

Posts:2,399
Registered: 10/26/00
Re: Splitting really huge XML files(50 GB avg size)  
Jul 15, 2004 2:28 PM (reply 28 of 31)



 
JFS supports 512 terabits w/512b blocks -> 4 petabytes w/4kb blocks
Reiserfs (a very good choice) supports 16 Terabytes (maybe OS X is using this?)
 

Posts:6
Registered: 3/31/04
Re: Splitting really huge XML files(50 GB avg size)  
Jul 15, 2004 5:01 PM (reply 29 of 31)



 
Hi,
My data changes from time to time so.... i have to actually parse to compile.
thanks,
 
This topic has 31 replies on 3 pages.    « Previous | 1 | 2 | 3 | Next »