Double Code Signing Greatly Increases Build Time

[Sheri_Steeves]

Hello,

I've just started migrating my builds from X3 to X5 and have hit a snag.

In X3 I was using the Authenticode section in the Project options to sign my install and as well to sign all executables in the install.

With X5 and the double signing I have encountered a huge jump in the time it takes to build my install.

Here is what I have found:

In X3, with Authenticode enabled for EXE

Sign all exectables (checkmark): 4 min 18 sec

In X5, with Authenticode enabled for EXE

Do not sign executables (no mark): 2 min 45 sec
Double sign all files not already signed (square): 16 min
Sign all executables (checkmark): 26 min

In X5, with Authenticode Unchecked : 1 min 45

My PFX is SHA-256 compatible, and I am using the http://timestamp.comodoca.com/rfc3161 timestamp URL.

Why the HUGE jump in build time? Is it just that the signing takes so much longer? Is it the time stamp server we are using? Do I have something set incorrectly?

This is a rather large issue for me as what used to take 4 minutes now takes 26. As I build multiple installs for each product (trial, versioned download, custom installs for clients) this drastically increases the time it takes me to build the installs. For example, to build a trial and a versioned download used to take me 8 minutes and will now, with the same settings, take 52minutes.

If I change my signing to sign only the files that are not already signed, my build time still increases by 12 minutes per build meaning what used to take me 8 minutes will now take 32 minutes.

Any ideas why?

Thanks,

Sheri

[FrancescoT]

Dear Sheri,

for sure the double signature process should take more time, but it should not take that much.

I don't believe your problem can be caused by the Timestamp URL, I use the same "daily" and never had any similar issue.

I suggest you to verify if any active Anti Virus may be involved with your problem. For example, it may be possible that the AV operates a sort of verification around the "signature process", it may verify the passed TimeStamp URL or the IA binaries that execute the process.

If this is the case ...I don't know how many files your projects has to sign, consider that this verification gets repeated each time the "signature process" tries to sign each single file.

Furthermore, also consider that the process that verifies if a file is already signed takes some additional time. Not that much for a single file of course, but At the end the sum of all the various verifications has to be multiplied with the number of files effectively involved.

Personally I can tell you that I never sign my setup files during the setup creation and I only leave IA to sign the generated setup package.
I Always sign all my files when these get compiled (from Visual Studio via build event) and this due the fact, that I want that the marked Timestamp exactly matches with the file creation date. If you have a chance to check, you will see that this approach is widely used by all most popular software vendors, even because this can "reduce" possible AV problems/conflicts when the file gets distributed or executed.

Hope this helps you.

Regards

[Sheri_Steeves]

Francesco,

Thanks for the reply. I will look into the possible anti-virus interference as I do have A/V running on the build machines.

I'm not sure at what point, or why, we decided to sign all the exe's through IA as we do sign our EXEs and DLLs through Visual Studio when building. It may have had something to do with false positives from Virus Checkers at some point, can't say for sure.

I did run some command line timing tests with just the sign tool on my machine, using the older SHA1 SignTool.exe and the newer SHA256 SignTool.exe from X5.

Unsigned exe, sha1 old timestamp server : 1s
Unsigned exe, sha1: 1s
Unsigned exe, sha256: 3s-4s
Add sha256 to sh1 signed exe: 3s
Double sign file already signed sha1+sha256 : 4s

My install tree has 56 executables and 174 dlls, at 4s/signing this is 920s - about 15 minutes but full re-sign build in IA is 26min
Looking at the build log when I am only signing if they are not signed: 137 files sign/double signed * 4s = 548/60 = ~9 min, overall process is 16 min.

I'll look into the A/V and followup. Thanks as always for your quick response.

Sheri

[Sheri_Steeves]

Francesco,

Just a followup on what I have found since yesterday.

I don't think it is antivirus as I disabled it and the install build took just as long.

I did some more timing tests and the results point to the time difference being the "http://timestamp.comodoca.com/rfc3161" time stamp server. This I know you have no control over. Do you know of any other server that does both sha1 and sha255 timestamping off the same server?

TimeStamp Server: http://timestamp.verisign.com/scripts/timstamp.dll
Unsigned exe, sha1: 1s

TimeStamp Server: http://timestamp.comodoca.com/rfc3161
Unsigned exe, sha1: 1s
Unsigned exe, add sha256: 3s-4s
Add sha256 to sh1 signed exe: 3s
Double sign file already signed sha1+sha256 : 4s

TimeStamp Server: http://sha1timestamp.ws.symantec.com/sha1/timestamp
Unsigned exe, sha1: 1s

TimeStamp Server: http://sha256timestamp.ws.symantec.com/sha256/timestamp
unsigned exe, add sha256: 1s
Add sha256 to sh1 signed exe: 1s
Double sign file already signed sha1+sha256 : 1.5s

At this point I think I'll need stop using the Authenicode tab, and use the symantec timestamp server. I'll add the 256 signing to my Visual Studio builds (which I needed to do anyway), adjust my install build scripts to manually sign the few EXE and DLL's that I need to sign after the build process (most of them are already signed, some are special), and possibly move the signing into the Build Events.

I'm going to take your suggestion of not signing everything - I honestly don't recall why were doing this anyway, unless it was just extra security to make sure everything was signed and we hadn't missed one during the build or had it fail to sign in the build process.

Thanks,

Sheri

[FrancescoT]

Dear Sheri,

first of all if you are using a commercial AV, very often is not enough to just disable it. This must be fully uninstalled in order to verify if this can be the problem. Unfortunately it doesn't exist any other possible way to check AV interferences.

Personally, I use exclusively the free Microsoft Essentials on my machines. It produces some interferences anyay, but these are really acceptable.

To return back to the matter, please consider that InstallAware doesn't use any trick to sign your files. The Whole process is entirely based on the Microsoft Signtool.exe (within the Authenticode sub-folder of the IA install directory). Alternatively the Signtool is available as part of the Windows SDK.
https://msdn.microsoft.com/en-us/library/windows/desktop/aa387764(v=vs.85).aspx

In addition, an InstallAware Code Signing Command Line tool is also available under the tools folder within the IA install menu (eg. start -> All Programs -> InstallAware X5).

Of course, the same identical Signtool can be called from Visual Studio as well.

That said and in case you want to continue your measures, it would be interesting to know how long it takes the sign process when signing a single file via signtool. At any rate and whatever will be the results, this should be quite identical as when signing from IA or VS. Of course, when signing from IA or VS you have to exclude all the extra jobs which are not effectively part of the signing process.

Regards

[Sheri_Steeves]

Francesco,

Sadly, I have no control over the A/V I use, company policy and controlled by server.

I didn't think IA was doing anything more than using the signtool. The timing results in my previous post were from using the IA signtool.exe to sign a single file via a batch script from a command prompt using different algorithms and timestamp servers. In most cases I started with an unsigned file, unless I was testing adding SHA256 to SHA1, or resigning an already signed file.

What I seemed to find here was that the comdoco server took 3s longer than the symantec server. I am in Canada, so perhaps location is an issue?

My batch file command were similar to below:

echo ----- START TIME IA SIGN TOOL SIGN sha1 rfc3161 symantec %TIME%
signtool.exe sign /d "%1" /p password /f "path to pfx" /tr "http://sha1timestamp.ws.symantec.com/sha1/timestamp" /td sha1 " .\Test.exe"
echo ----- END TIME IA SIGN TOOL sha1 rfc3161 symantec %TIME%
echo.

echo ----- START TIME IA SIGN TOOL SIGN sha1+sha256 rfc3161 symantec %TIME%
signtool.exe sign /d "%1" /f "path to pfx" /as /fd sha256 /tr "http://sha256timestamp.ws.symantec.com/sha256/timestamp" /td sha256 /p password .\Test.exe"
echo ----- END TIME IA SIGN TOOL sha1+sha256 rfc3161 symantec %TIME%
echo

I am able to work ahead with what I have discovered and keep my install build speed to a minimum; which was my main problem.

Sheri