Java 7 / Java 8 Large File Read and Run times Compared ...

[David]

Firstly ... a Java program to generate a big file of random integers ...

Code: [Select]

/**
 * Reports the number of seconds the computer spent
 * creating a file of 3,000,000 random integers,
 * 10 integers per line
 * with a single space in between each integer on each line.
 * (Note! There is NO space output following the 10th integer on each line.)
 *
 * @version 2017-12-06
 * @author dwz
 */
 

import java.nio.file.Paths;
import java.nio.file.Files;
import java.nio.charset.Charset;

import java.io.BufferedWriter;
import java.io.IOException;

import java.util.Random;


public class TimedWriteLine {
   
    static final String FNAME = "MyRandomInts.txt";
    static final int NUM_NUMBERS = 3_000_000;


    public static void main( String[] args ) {

        testItOut();
    }


    static void testItOut() {

        System.out.println( "Getting/filing " + NUM_NUMBERS + " random integers." );

        long startTime = System.currentTimeMillis();
        Random randGen = new Random();

        try( BufferedWriter fout = Files.newBufferedWriter( Paths.get( FNAME ),
                Charset.defaultCharset() ) ) {
       
            for( int i = 0; i < NUM_NUMBERS; ) {
                String valStr =  Integer.toString( randGen.nextInt(NUM_NUMBERS) );
                fout.write( valStr );
                ++ i;
                if( i % 10 == 0 ) fout.newLine();
                else fout.write( " ", 0, 1 );
            }
        } catch( IOException ex ) {
            System.out.println( ex );
        }
     
        long endTime = System.currentTimeMillis();
        System.out.println();
        System.out.println( "Run time in seconds was: " + (endTime - startTime) / 1000.0 );
    }
}

[David]

Now a timed  'readline' method ... reading this large file.

Code: [Select]

/**
 * Reports the number of seconds the computer spent
 * reading 300,000 lines, line by line,
 * and then parsing out the 10 integers on each line
 * adding each integer, as parsed out, to a growing List < Integer >
 *
 * @version 2017-12-06
 * @author dwz
 */
 

import java.nio.file.Paths;
import java.nio.file.Files;
import java.nio.charset.Charset;

import java.io.BufferedReader;
import java.io.IOException;

import java.util.ArrayList;
import java.util.List;


public class TimedReadLine {
   
    static String FNAME = "MyRandomInts.txt";
    static List < Integer > nums = new ArrayList <> ();

    public static void main( String[] args ) {
       
        long startTime = System.currentTimeMillis();
        testItOut();
        long endTime = System.currentTimeMillis();
        System.out.println();
        System.out.println( "Run time in seconds was: " + (endTime - startTime) / 1000.0 );
        System.out.println( "There are " + nums.size() + " numbers in the List of nums." );
    }

    static void testItOut() {

        try( BufferedReader fin = Files.newBufferedReader( Paths.get( FNAME ),
                Charset.defaultCharset() ) ) {
            String line = null;
            while( ( line = fin.readLine() ) != null ) {
                processLine( line );
            }
        } catch( IOException ex ) {
            System.out.println( ex );
        }
    }
   
    static void processLine( String line ) {

        String[] ary = line.split( " " );
        try {
            for( String val : ary )
                nums.add( Integer.parseInt(val) );
        } catch( Exception ex ) {
            System.out.println( ex );
        }
    }
}

[David]

And now a timed 'stream' file read method ...

Code: [Select]

/**
 * Reports the number of seconds the computer spent
 * reading 300,000 lines, via a Java 8 type Stream object,
 * and using its forEach method,
 * to then parsing out the 10 integers on each line,
 * adding each integer, as parsed out, to a growing List < Integer >
 *
 * @version 2017-12-06
 * @author dwz
 */
 

import java.nio.file.Paths;
import java.nio.file.Files;
;
import java.io.IOException;

import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;


public class TimedReadStream {
   
    static String FNAME = "MyRandomInts.txt";
    static List < Integer > nums = new ArrayList <> ();

    public static void main( String[] args ) {
       
        long startTime = System.currentTimeMillis();
        testItOut();
        long endTime = System.currentTimeMillis();
        System.out.println();
        System.out.println( "Run time in seconds was: " + (endTime - startTime) / 1000.0 );
        System.out.println( "There are " + nums.size() + " numbers in the List of nums." );
    }

    static void testItOut() {

        try( Stream< String > myLines = Files.lines( Paths.get( FNAME ) ) ) {
            myLines.forEach( line -> processLine( line) );
        } catch( IOException ex ) {
            System.out.println( ex );
        }
    }
   
    static void processLine( String line ) {

        String[] ary = line.split( " " );
        try {
            for( String val : ary )
                nums.add( Integer.parseInt(val) );
        } catch( Exception ex ) {
            System.out.println( ex );
        }
    }
}



Below , are some typical result times on my Windows 7 OS laptop , right after a computer 'reboot',
for 3 successive runs of reading with the new Java 8 Stream read methods,
followed by 3  sets of reading 'line by line' ...

then the computer was rebooted and the order of the sets was reversed with the read 'line by line' at the start ...

NOTE the MUCH longer read times for the first file read, in a set of repeats to read that file, when no part of the file is yet in cache memory right after the computer has been rebooted!



Computer was 're-booted' ... then these tests were done:

Run time in seconds was: 7.704
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.846
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.832
There are 3000000 numbers in the List of nums.
Stream ...


Run time in seconds was: 2.035
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.808
There are 3000000 numbers in the List of nums.

Run time in seconds was: 2.075
There are 3000000 numbers in the List of nums.
Line ...


Press any key to continue . . .



Run time in seconds was: 2.149
There are 3000000 numbers in the List of nums.

Run time in seconds was: 2.043
There are 3000000 numbers in the List of nums.

Run time in seconds was: 2.106
There are 3000000 numbers in the List of nums.
Stream ...


Run time in seconds was: 1.878
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.918
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.96
There are 3000000 numbers in the List of nums.
Line ...


Press any key to continue . . .



Computer was 're-booted' ... then these tests were done:

Run time in seconds was: 3.37
There are 3000000 numbers in the List of nums.

Run time in seconds was: 2.081
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.981
There are 3000000 numbers in the List of nums.
Line ...


Run time in seconds was: 2.1
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.997
There are 3000000 numbers in the List of nums.

Run time in seconds was: 2.005
There are 3000000 numbers in the List of nums.
Stream ...


Press any key to continue . . .




Run time in seconds was: 1.919
There are 3000000 numbers in the List of nums.

Run time in seconds was: 2.1
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.982
There are 3000000 numbers in the List of nums.
Line ...


Run time in seconds was: 2.056
There are 3000000 numbers in the List of nums.

Run time in seconds was: 2.013
There are 3000000 numbers in the List of nums.

Run time in seconds was: 1.981
There are 3000000 numbers in the List of nums.
Stream ...


Press any key to continue . . .