18. IO Benchmark


I/O access times play a crucial role in the overall performance of a database. It is important to estimate the performance before committing to a specific database. To help you with this we introduce IO Benchmark tool with the following functionality:
1.    Measuring the actual I/O performance of a system as seen by db4o
2.    Simulating the behaviour of a slower system on a faster one
IO benchmark consists of db4o_bench.jar, with the source code in the db4otools project / com.db4o.bench package.
The code can be compiled with JDK 1.3 and higher.
Benchmark runs in 3 stages:
1.    Run a target application and log its I/O access pattern using LoggingIoAdapter:
// runTargetApplication
// Stage 1: running the application to record IO access
System.out.println("Running target application ...");
// Any custom application can be used instead
new CrudApplication().run(itemCount);

2.    Replay the recorded I/O operations once to prepare a database file.
// prepareDbFile
// Stage 2:Replay the recorded IO to prepare a database file
System.out.println("Preparing DB file ...");
deleteFile(_dbFileName);
IoAdapter rafFactory = new RandomAccessFileAdapter();
IoAdapter raf = rafFactory.open(_dbFileName, false, 0, false);
// Use the file with the recorded operations from stage 1
LogReplayer replayer = new LogReplayer("simplecrud_" + itemCount + ".log", raf);
try {
    replayer.replayLog();
} catch (IOException e) {
    exitWithError("Error reading I/O operations log file");
} finally {
    raf.close();
}

This step is necessary to ensure that during the grouped replay in the next step, none of the accesses will go beyond the currently existing file.

3.    Replay the recorded I/O operations a second time. Operations are grouped by command type (read, write, seek, sync), and the total time executing all operations of a specific command type is measured. Grouping is necessary to avoid micro-benchmarking effects and to get time values above timer resolution.
We divide the numbers collected in stage 3 by the respective number of operations and we calculate the  average time a particular command takes on the given system.
// runBenchmark
// Stage 3: Replay the recorded IO grouping command types
System.out.println("Running benchmark ...");
deleteBenchmarkResultsFile(itemCount);
PrintStream out = new PrintStream(new FileOutputStream(resultsFileName(itemCount), true));
printRunHeader(itemCount, out);
// run all commands: READ_ENTRY, WRITE_ENTRY, SYNC_ENTRY, SEEK_ENTRY
for (int i = 0; i < LogConstants.ALL_CONSTANTS.length; i++) {
    String currentCommand = LogConstants.ALL_CONSTANTS[i];
    benchmarkCommand(currentCommand, itemCount, _dbFileName, out);    
}
out.close();
deleteFile(_dbFileName);
deleteCrudLogFile(itemCount);




    18.1. Benchmark Application

    IO Benchmark can be used with any custom application. Here we will look at it on an example of a simple CRUD application.
    /* Copyright (C) 2007  db4objects Inc.  http://www.db4o.com */
    package com.db4o.f1.chapter10;
    import java.io.*;
    import com.db4o.*;
    import com.db4o.bench.logging.*;
    import com.db4o.config.*;
    import com.db4o.io.*;
    /**
    * Very simple CRUD (Create, Read, Update, Delete) application to
    * produce log files as an input for I/O-benchmarking.
    */
    public class CrudApplication {
        
        private static final String DATABASE_FILE = "simplecrud.db4o";
        
        public void run(final int itemCount) {
            Configuration config = prepare(itemCount);
            create(itemCount, config);
            read(config);
            update(config);
            delete(config);
            deleteDbFile();
        }
        private Configuration prepare(int itemCount) {
            deleteDbFile();
            RandomAccessFileAdapter rafAdapter = new RandomAccessFileAdapter();
            IoAdapter ioAdapter = new LoggingIoAdapter(rafAdapter, logFileName(itemCount));
            Configuration config = Db4o.cloneConfiguration();
            config.io(ioAdapter);
            return config;
        }
        private void create(int itemCount, Configuration config) {
            ObjectContainer oc = open(config);
            for (int i = 0; i < itemCount; i++) {
                oc.store(Item.newItem(i));
                // preventing heap space problems by committing from time to time
                if(i % 100000 == 0) {
                    oc.commit();
                }
            }
            oc.commit();
            oc.close();
        }
        
        private void read(Configuration config) {
            ObjectContainer oc = open(config);
            ObjectSet objectSet = oc.query(Item.class);
            while(objectSet.hasNext()){
                Item item = (Item) objectSet.next();
            }
            oc.close();
        }
        
        private void update(Configuration config) {
            ObjectContainer oc = open(config);
            ObjectSet objectSet = oc.query(Item.class);
            while(objectSet.hasNext()){
                Item item = (Item) objectSet.next();
                item.change();
                oc.store(item);
            }
            oc.close();
        }
        private void delete(Configuration config) {
            ObjectContainer oc = open(config);
            ObjectSet objectSet = oc.query(Item.class);;
            while(objectSet.hasNext()){
                oc.delete(objectSet.next());
                // adding commit results in more syncs in the log,
                // which is necessary for meaningful statistics!
                oc.commit();    
            }
            oc.close();
        }
        private void deleteDbFile() {
            new File(DATABASE_FILE).delete();
        }
        private ObjectContainer open(Configuration config) {
            return Db4o.openFile(config, DATABASE_FILE);
        }
        public static String logFileName(int itemCount) {
            return "simplecrud_" + itemCount + ".log";
        }
        
    }

    Please, pay attention to prepare method, which configures the use of LoggingIoAdapter - this is the only change that is required on your application's side to make it available for benchmarking. LoggingIoAdapter will ensure that all IO access operations will be logged to the specified file. This information can be used later (stage 2 and 3 of the benchmark) to replay the application's database interaction and measure performance for this specific pattern on different environments.


    18.2. Benchmark Example

    You can try to run the benchmark immediately on our sample application. We use a very small number of objects (10) for this example to make it faster:
    // runNormal
    printDoubleLine();
    RandomAccessFileAdapter rafAdapter = new RandomAccessFileAdapter();
    IoAdapter ioAdapter = new LoggingIoAdapter(rafAdapter, "test.log");
            
    System.out.println("Running db4o IoBenchmark");
    printDoubleLine();
    // Run a target application and log its I/O access pattern
    runTargetApplication(_objectCount);
    // Replay the recorded I/O operations once to prepare a database file.
    prepareDbFile(_objectCount);
    // Replay the recorded I/O operations a second time.
    // Operations are grouped by command type (read, write, seek, sync),
    // and the total time executing all operations of a specific command type is measured.
    runBenchmark(_objectCount);

    You can use the above mentioned sequence of operations to benchmark any db4o application:
    - make sure that your application uses LoggingIoAdapter for the benchmark run;
    - modify runTargetApplication method to call your application.
    The ns (nanosecond) values are our benchmark standard for the respective operations.  Smaller numbers are better.
    Note: It may be possible, that you get some zero values for time elapsed, and therefore infinity for operations per ms. This can occur if your machine is fast enough to execute all operations under 1 ms. To overcome this you can run the run.benchmark.medium target which operates with more objects and takes longer to complete.


    18.3. Delayed Benchmark

    IO Benchmark provides you with another helpful feature - delayed benchmark. It can be used to benchmark target devices (with a slow IO) on your development workstation. In order to do so you will need to run benchmark on your workstation and on target device, copy the result files to some folder on your workstation and run delayed benchmark. The delayed benchmark will use benchmark results files to analyze how much slower is the device than the workstation: a special delays will be introduced for each operation. Let's look at an example.
    You should have the benchmark results from the previous example run. If you did not run the previous example, please, do it now. An example benchmark result data from a target device is already prepared and will be saved into db4o-IoBenchmark-results-10-slower.log. Workstation and target device results are defined in the benchmark example as:
    private static final String _resultsFile2 = "db4o-IoBenchmark-results-10-slower.log";
    private static final String _resultsFile1 = "db4o-IoBenchmark-results-10.log";

    Note, that the order of files does not matter - the delays will be calculated to match the slower one.
    // runDelayed
    printDoubleLine();
    System.out.println("Running db4o IoBenchmark");
    printDoubleLine();
    // Write sample slow data to the test file
    prepareDelayedFile(_resultsFile2);
    // calculate the delays to match the slower device
    processResultsFiles(_resultsFile1, _resultsFile2);
    // Run a target application and log its I/O access pattern
    runTargetApplication(_objectCount);
    // Replay the recorded I/O operations once to prepare a database file.
    prepareDbFile(_objectCount);
    // Replay the recorded I/O operations a second time.
    // Operations are grouped by command type (read, write, seek, sync),
    // and the total time executing all operations of a specific command type is measured.
    runBenchmark(_objectCount);

    Now you are supposed to get slower results (be patient - it can take a while). The example in this tutorial might be a bit confusing as it operates on very little amount of objects (thus time adjustments can be largely rounded), and the slow results file contains test data, which might not be realistic for any real device environment. There are some other pitfalls that you must remember when using delayed benchmark:
    - Delayed benchmark can only work if all measured operations (read, write, seek, sync) are slower on one device and faster on another. If that is not the case an error will be returned.
    - There is a minimum delay that can be achieved (a delay of processing operation). If your target device requires a smaller than minimum delay - the benchmark results will be inaccurate. You can try to improve the mesurements by running on a bigger amount of objects.
    For any questions and improvement suggestions, please, use our forums or Jira.  





    www.db4o.com