2002-02-06: The IDLSpecII benchmark is obsolete, and is no longer accepting entries. It will hopefully be replaced by an IDLSpecIII in the not too distant future. IDLSpecIII will require an updated benchmark, possibly including a 3D Graphics suite. If you have interest in contributing to the development of this benchmark, contact JD Smith.

Welcome to the IDLSpecII benchmark page. The purpose of this survey is to collect and fairly compare the performance of RSI's IDL software on a variety of architectures. You can skip to the performance plot now. This survey offers improvements over its predecessor (IDLSpec) in several ways:

• Supports accurate measurement of the fastest systems using the RSI supplied time_test3 and graphics_times3 routines.
• Calculates a single quantity, IDLSpecMark, which can be used to compare entries.
• Considers the graphics performance on machines with differing video bit depths (8 and 24) separately.
• On unix systems, attempts to automatically email the results... all you have to do is run it! (Windows and Macintosh users still need to email the record by hand).
• Automatically processes emailed entries and updates database and figure nightly.
• Clickable performance comparison plot with popup information on individual entries (on JavaScript enabled browsers), indexed into the survey results.

IDLSpec II now contains 160 entries.

How to Enter

Make sure you are using IDL version 5.2 or greater (5.2.1 under Linux). In order to provide useful comparisons, older versions of IDL are not supported in IDLSpecII.

To enter the survey and have your systems compared against others, simply download (by shift-clicking on the link, for instance) one of following two files:

• is2_52.sav for IDL versions 5.2 and 5.2.1.
• is2_53.sav for IDL version 5.3.

IDL> restore,'pathtofile/is2_vers.sav' & idlspec2

Details

IDLSpecII uses the RSI-supplied suite of time testing routines collectively called "time_test" to generate the performance indices and the IDLSpecMark. The arithmetic and geometric means for the processing and graphics test, along with a single I/O test are utilized. The IDLSpecMark is the weighted mean of the I/O test index, and the arithmetic mean of the processing test results. Each test result time is first normalized to a reference machine (see below). The IDLSpecMark places more emphasis (roughly twice) on processing power than on I/O performance, and does not include graphics performance at all (which depends on too many parameters). As always, keep in mind that a single number cannot encapsulate the performance characteristics of any system.

N.B. As of 5/23/00 the CPU and Graphics indices and the IDLSpecMark were changed to use only the arithmetic mean of test results. The IDLSpecMark still gives CPU performance twice as much weight as I/O performance. This change was necessary because the fastest machines often complete individual tests with zero time, corrupting the geometric mean. A more computationally intensive set of benchmarks is planned.

The graphics measurement does not perform any 3D tests, which would introduce even more complexity since the support of OpenGL 3D drawing primitives in hardware and/or software varies widely in support among machines, video equipment, and OS's.

Some of you might wonder why idlspec2 is distributed as a bytecode compiled binary-only file (".sav" file). Nothing done by idlspec2 is proprietary or secret. It simply uses the graphics_times3 and time_test3 routines along with the I/O test from time_test3 extracted to keep these results separate. The reason for distributing in binary is that idlspec2 includes a hash function which produces a key included in the record set. This key is automatically checked against the other data in the record to ensure it was not modified or garbled in transit. This will prevent the database from becoming polluted, either accidentally, or maliciously, and will ensure the integrity of the entries. If you are interested in seeing a copy of the source code with the hash generating code removed, contact me.

Performance Plot

The figure below is a scatter plot of the relative processing and I/O speeds of all entrants. Fastest entries are in the upper right. The performance indices consist of the relevant arithmetic means normalized to a reference system: PentiumII 400MHz 128MB Linux 2.0.36. The video cards to which the graphics results have been referenced are: 8-bit: Diamond Fire GL 1000 Pro 8MB; 24-bit: ATI Rage LT Pro 8MB. These systems by definition have all performance indices equal to one, so that for example an entry with I/O performance index of 2 has I/O speed approximately twice the reference system.

The extent of the symbols scales linearly with normalized graphics index, to lie between a fixed minimum and maximum size. A system with graphics performance halfway between two others has their average symbol size. Thus the plotted graphics information allows you to compare relative, not absolute, graphics performance, which is available in the form of the computed graphics indices. Circles correspond to systems with 8-bit graphics, while systems with bit-depths higher than 8 are plotted with squares. Be sure to compare graphics results only for systems of the same bit-depth! Different OS:Architecture combinations are plotted in different colors as indicated by the legend below the plot.

Moving the mouse over individual entries pops-up a synopsis of the record on JavaScript-enabled browswers. Clicking on an individual plot symbol takes you to the corresponding entry in the sorted record file.