Fast Notes view reading via C API: A comparison C vs. Notes.jar
Karsten Lehmann 31 March 2010 22:09:36
Bad day for a blog posting - this is no April fool's joke. :-)About 1,5 weeks ago, Nathan Freeman wrote a blog posting about the hidden NAPI plugin in Lotus Notes 8.5.1 that IBM uses to speed up the XPages framework. NAPI is an unsupported and incomplete set of Java classes that use JNI technology (Java Native Interface) to leverage functions of the Notes C API, for example to read data from a Notes view.
According to Nathan's speed race video, the NAPI classes are much faster than using the classic Notes Java API inside the Notes.jar archive. To read 15.000 view entries, NAPI takes 1 second where Notes.jar is running for 6 seconds - a pretty impressive result.
Last week, inspired by his comparison, we decided to develop our own C/JNI code. Of course, we like nothing more than writing C and it was a question of honour not to use IBM's NAPI plugin. :-)
No, just kidding. Our use case was a bit different. NAPI is an Eclipse plugin, but we needed a solution that also runs in standalone Java applications.
A few days of fun with NIF'ty things like NIFOpenCollection, NIFGetCollectionData or NIFReadEntries later, we now have our own working solution.
The result
We took Nathans test database with 15.000 documents to measure the performance of the different solutions. Our test code is executed from the Eclipse IDE as a small standalone application, we did not build an Eclipse plugin for it yet.
The view looks like this:
Three columns with text, one hidden sort column with the severity as number.
Traversing the view and reading all column values using Notes.jar takes 2.8 seconds on our desktop test machine (Intel Core2Duo, 3 GHz, 4 GB RAM. WinXP VM in Vista)
Please note that our test code is running exclusively in the JVM. This may explain the gap to Nathan's 6 seconds, his four test cases run in parallel which probably slows down I/O throughput, causes synchronization losses in the API and splits the CPU power.
Speed of our C/JNI solution
For the same scenario, the C/JNI approach only takes 0.93 seconds. You can even save more time by reducing the columns to be read. In contrast to Notes.jar, where you just have
ViewEntry.getColumnValues(), which decodes all columns in a row, by using the C API you have to decode each column separately (e.g. to convert Notes LMBCS strings to Java's Unicode format). For example, if you decide to only decode one column, let's say the last one, the duration drops to 0,8 seconds. If you are just interested in the sort order of documents, you might only read the Note ID and skip the column decoding, which results in 0,7 seconds.
Drawbacks
0,93 seconds instead of 2,8 sounds pretty dramatic, but we need to keep in mind that we are not reading 10 or 100 view entries, but 15.000. So for normal lookup operations, this solution is probably an overkill. Especially because in a hybrid scenario (where most operations are done in Notes.jar, but view reading in C/JNI) we need to create a second session and may open the database and view twice, which obviously consumes some time.
However, it looks like there is a good use case for the JNI approach in scenarios where a lot of processes access Notes data in parallel, for example on a Domino web server or in the Lotus Notes Standard Client with a lot of plugins that use Notes APIs.
In those cases, Notes.jar turns out to be a performance bottleneck, because it does not directly wrap the C API functions, but is based on the "Lotus Software Extension Backend (LSXBE)", which does not handle multithreading very well from a performance perspective.
Click here to read the second article about C API vs. Notes.jar - how to speed up view reading by the factor 335.
- Comments [4]