A new benchmark analyzed the compression efficiency and installation speeds of one open source and two leading commercial installation platforms, finding that the commercial InstallAware X2 platform defeated both the commercial InstallShield 2014 platform and the open source WiX 3.9 platform; on both speed and size.
Initial Approach: The InstallAware X2 Program Files folder, containing 15,985 files, 2,860 folders, and 14.7 GB of data, was to be installed using WiX 3.9, InstallAware X2, and InstallShield 2014 – from a single, self-extracting executable suitable for online distribution. This initial approach failed as follows:
o InstallAware X2 was able to build a Native Engine setup that could install this complete folder.
o The Heat tool in the WiX 3.9 toolkit crashed while collecting file data from the Plug-Ins subfolder, and didn’t crash but displayed warnings and exceptions while collecting file data from the Runtimes subfolder (possibly due to extra-long file paths which are unrecognized by WiX).
o The InstallShield 2014 IDE crashed while building a Windows Installer based setup file. The output size of the InstallShield 2014 IDE while building an InstallScript based setup file was in excess of 4GB, which cannot be run under Windows due to Windows’s own technical limitations.
Since WiX 3.9 could not successfully build a setup, and since InstallShield 2014 could only build a setup that couldn’t actually run on Windows, the initial approach had to be modified to enable a comparison.
Modified Approach: A small subset of files inside the InstallAware X2 Program Files folder were installed – all of the nine 7ZIP archives inside the Runtimes subfolder, together with the sqlexpress2005bin_x64 and the sqlexpress2008bin_x64 subfolders of the Runtimes folder. This modified approach succeeded as follows:
o InstallAware X2 built a setup in 19 minutes and 27 seconds, weighing in at 430,673 KB, self-extracting in 45 seconds, and installing in 3 seconds.
o WiX 3.9 built a setup in 4 minutes, weighing in at 1,397,208 KB, self-extracting in 14 seconds, and installing in 42 seconds.
o InstallShield 2014 crashed again while building a Windows Installer based setup file. InstallShield 2014 built an InstallScript based setup file in 3 minutes and 41 seconds, weighing in at 1,179,986 KB, self-extracting in 6 seconds, and installing in 1 minute and 6 seconds.
All tests were conducted on a PC with 16 GB of RAM, powered by an AMD FX 6300 8-core CPU.
Commentary: InstallAware X2 was the slowest to build and self-extract, but installation was an order of magnitude faster after self-extraction, and the produced setup was the smallest of the three. WiX 3.9 produced the largest setup of the three, and InstallShield 2014 was the slowest to install among the three. InstallShield 2014 was the fastest to self-extract and the fastest to build among the three.
Conclusion: InstallAware X2 installed 50% faster than InstallShield 2014 and 17% faster than WiX 3.9. InstallAware X2 compressed 174% smaller than InstallShield 2014 and 224% smaller than WiX 3.9. Software installation developers are encouraged to reproduce and validate the results on their own systems. All of the setup projects used in the tests above are available for immediate download at http://www.installaware.com/acidtest.asp. Developers are strongly encouraged to experiment with their own apps and report their findings to firstname.lastname@example.org.