Call DLL Function on 64bit OS - Second call hangs

[tj]

We have identified a repeatable situation in which the first call to our dll succeeds and the second call hangs. The dlls calls work just fine on a 32bit OS but exhibit this behavior on a 64bit OS.

We have created a small repeatable application to illustrate the problem. The app is commented to explain what we are seeing.

I will attach the InstallAware app and supporting files for details on how to reproduce the issue after I make this initial post. I had problems uploading the zip file.

[tj]

I couldn't attach these in the original post

File Attached:

DLLIssue.zip

[tj]

This is the IA App... I couldn't attach these files with the original post.

File Attached:

IADllTest.zip

[tj]

bump?

This continues to be an issue for us. I'm surprised others haven't seen the same issue yet.

Again, we packaged up a small, repeatable installation and custom dll to illustrate the problem. Unfortuately, we are going to have to resort to a work around, passing values between a text file, if the response from the dlls does not work on the 64 bit boxes.

We'd really appreciate any info or feedback from others working with custom dlls on 64bit boxes. Again, we have/had to problems on the 32 bit boxes with the same dll.

regards

[MichaelNesmith]

Did you try setting 32 bit installation mode before calling the DLL and then restoring the current setup mode after calling the DLL?

[tj]

I just tried setting the 64bit modes explicitly using the Set 64bit Mode command. The little sample installer we provided above continues to hang on a 64bit OS but works fine on 32bit even with these commands in place.

Can anyone else verify this behavior? I saw a slightly similar post:
http://www.installaware.com/forum/viewt ... LED#p19693

I can't say that I think they are the same issue but it sounded close considering it fails the SECOND time through.

Unfortunately, we don't have a good work around. This dll is critical for our setup because it saves encrypted credentials into a custom config file for use when upgrading/modifying the setup.

Any other recommendations?

[MichaelNesmith]

I don't see anything in your projects about setting 32/64 bit mode.

Also, based on the comments in your code about the "/clr" switch changing things, it might be an issue with your compiler or project settings itself.

I would suggest you to display message boxes from your own DLL to make sure it is loaded successfully when invoked using Call DLL Function.

It will also help if you can replicate the same issue with native Windows DLL calls, such as CopyFileA in kernel32.dll - that would definitively demonstrate the issue is with InstallAware, and not your DLL. Right now, the issue could be with your DLL.

[mills]

tj,

Are you still experiencing problems with Call DLL Function on Win 2k8 x64?

I have tried to reproduce the issue using your project to no avail. It's possibly a condition in your environment that is the source of the problem...

If you're still having problems, please test with a system dll like Michael suggested and it would also be good to know what version of InstallAware you're using.

[mills]

This post is related:
http://forums.installaware.com/viewtopic.php?f=2&t=5297

[tj]

Yes, we were still seeing this behavior on the 2008 R2 64bit platform specifically.

As a matter of fact, it became such a big issue, that we re-wrote the dll. We think the core of the issue is related to the fact that the dll simply had the /clr switch. We were using the .NET framework but, just having the /clr switch itself (WITH NO CALL) cause the problem. We removed the need to use .NET for our encryption/config files. We now write everything to a custom binary format using C++ alone.

Hope this helps. This issue was not easy to track down.

I'd be happy to answer any questions or provide more info for anyone troubleshooting this issue.
tj

[mills]

I just find it odd that the test project you provided exhibited no issues on a clean Win2k8 x64 install for me...

So, it sounds like something to do with .Net is involved here, then?

What version of InstallAware are you using? What version of .Net was being used in the test environment?

[tj]

Be sure it's R2 specifically.

We had many clients reproduce the error. Thus, the need for our change.

It definitely had something to do with the /clr switch. Even if we made NO CALL, that switch seemed to make all the difference.

I'm not sure we can send the actual dll and installer program... but, if I have a weekend moment, I'll try to package up another sample. That said, I think we just resurrected this using that same sample for our emergency rewrite as the base test. We JUST made this change about 3 weeks ago.

We are currently using 9.1.9010.2010
We saw this issue originally back on 8??
Our setup was happy as can be until people started installing on that R2 platform on x64 boxes.

[mills]

/clr switch info for my notes:
http://msdn.microsoft.com/en-us/library/k8d11d4s.aspx

What version of .Net are you testing with? What version on your development and testing machines?

I added some messageboxes to the test project you provided. I was getting interesting results in both XP (pictured right) and 2k8 (on the left):

dllCallResult.png (67.5 KiB) Viewed 80608 times

I did verify the function existed in the dll (through DLL Export Viewer), so not sure why I'm getting these results...

[tj]

We are running .Net 2.0 primarily but we have run this in environments with many other variations.
The problematic environment has always been the 2008 64bit R2 which usually has everything up to 3.5 intalled.

[mills]

What result are you getting for the failed call?