Page 1 of 1

Call DLL Function on 64bit OS - Second call hangs

Posted: Tue Jan 19, 2010 9:00 am
by tj
We have identified a repeatable situation in which the first call to our dll succeeds and the second call hangs. The dlls calls work just fine on a 32bit OS but exhibit this behavior on a 64bit OS.

We have created a small repeatable application to illustrate the problem. The app is commented to explain what we are seeing.

I will attach the InstallAware app and supporting files for details on how to reproduce the issue after I make this initial post. I had problems uploading the zip file.

The DLL Project

Posted: Tue Jan 19, 2010 9:09 am
by tj
I couldn't attach these in the original post

File Attached:

DLLIssue.zip

The IA app

Posted: Tue Jan 19, 2010 9:10 am
by tj
This is the IA App... I couldn't attach these files with the original post.

File Attached:

IADllTest.zip

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Mon Jun 07, 2010 12:50 pm
by tj
bump?

This continues to be an issue for us. I'm surprised others haven't seen the same issue yet.

Again, we packaged up a small, repeatable installation and custom dll to illustrate the problem. Unfortuately, we are going to have to resort to a work around, passing values between a text file, if the response from the dlls does not work on the 64 bit boxes.

We'd really appreciate any info or feedback from others working with custom dlls on 64bit boxes. Again, we have/had to problems on the 32 bit boxes with the same dll.

regards

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Mon Jun 07, 2010 4:09 pm
by MichaelNesmith
Did you try setting 32 bit installation mode before calling the DLL and then restoring the current setup mode after calling the DLL?

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Thu Jun 24, 2010 12:57 pm
by tj
I just tried setting the 64bit modes explicitly using the Set 64bit Mode command. The little sample installer we provided above continues to hang on a 64bit OS but works fine on 32bit even with these commands in place.

Can anyone else verify this behavior? I saw a slightly similar post:
http://www.installaware.com/forum/viewt ... LED#p19693

I can't say that I think they are the same issue but it sounded close considering it fails the SECOND time through.

Unfortunately, we don't have a good work around. This dll is critical for our setup because it saves encrypted credentials into a custom config file for use when upgrading/modifying the setup.

Any other recommendations?

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Thu Jun 24, 2010 6:44 pm
by MichaelNesmith
I don't see anything in your projects about setting 32/64 bit mode.

Also, based on the comments in your code about the "/clr" switch changing things, it might be an issue with your compiler or project settings itself.

I would suggest you to display message boxes from your own DLL to make sure it is loaded successfully when invoked using Call DLL Function.

It will also help if you can replicate the same issue with native Windows DLL calls, such as CopyFileA in kernel32.dll - that would definitively demonstrate the issue is with InstallAware, and not your DLL. Right now, the issue could be with your DLL.

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Tue Oct 19, 2010 2:15 pm
by mills
tj,

Are you still experiencing problems with Call DLL Function on Win 2k8 x64?

I have tried to reproduce the issue using your project to no avail. It's possibly a condition in your environment that is the source of the problem...

If you're still having problems, please test with a system dll like Michael suggested and it would also be good to know what version of InstallAware you're using.

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Tue Oct 19, 2010 2:15 pm
by mills

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Tue Oct 19, 2010 2:27 pm
by tj
Yes, we were still seeing this behavior on the 2008 R2 64bit platform specifically.

As a matter of fact, it became such a big issue, that we re-wrote the dll. We think the core of the issue is related to the fact that the dll simply had the /clr switch. We were using the .NET framework but, just having the /clr switch itself (WITH NO CALL) cause the problem. We removed the need to use .NET for our encryption/config files. We now write everything to a custom binary format using C++ alone.

Hope this helps. This issue was not easy to track down.

I'd be happy to answer any questions or provide more info for anyone troubleshooting this issue.
tj

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Tue Oct 19, 2010 2:49 pm
by mills
I just find it odd that the test project you provided exhibited no issues on a clean Win2k8 x64 install for me...

So, it sounds like something to do with .Net is involved here, then?

What version of InstallAware are you using? What version of .Net was being used in the test environment?

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Tue Oct 19, 2010 3:03 pm
by tj
Be sure it's R2 specifically.

We had many clients reproduce the error. Thus, the need for our change.

It definitely had something to do with the /clr switch. Even if we made NO CALL, that switch seemed to make all the difference.

I'm not sure we can send the actual dll and installer program... but, if I have a weekend moment, I'll try to package up another sample. That said, I think we just resurrected this using that same sample for our emergency rewrite as the base test. We JUST made this change about 3 weeks ago.

We are currently using 9.1.9010.2010
We saw this issue originally back on 8??
Our setup was happy as can be until people started installing on that R2 platform on x64 boxes.

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Tue Oct 26, 2010 2:27 pm
by mills
/clr switch info for my notes:
http://msdn.microsoft.com/en-us/library/k8d11d4s.aspx

What version of .Net are you testing with? What version on your development and testing machines?

I added some messageboxes to the test project you provided. I was getting interesting results in both XP (pictured right) and 2k8 (on the left):
dllCallResult.png
dllCallResult.png (67.5 KiB) Viewed 80618 times
I did verify the function existed in the dll (through DLL Export Viewer), so not sure why I'm getting these results...

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Mon Nov 01, 2010 1:54 pm
by tj
We are running .Net 2.0 primarily but we have run this in environments with many other variations.
The problematic environment has always been the 2008 64bit R2 which usually has everything up to 3.5 intalled.

Re: Call DLL Function on 64bit OS - Second call hangs

Posted: Mon Nov 01, 2010 10:23 pm
by mills
What result are you getting for the failed call?