Call DLL Function return result value

jsmith · Post by **jsmith** » Mon Dec 08, 2008 3:34 pm

I have a custom action DLL that I wrote in C++. The function I want to call in the DLL has a __stdcall signature and returns a UINT. What return type must I specify in InstallAware's "Call DLL Function" dialog to get the return code from my function? No matter what type I specify, the variable that I specify in the "Return result in variable:" text box is blank after the call to Call DLL Function.

jsmith · Post by **jsmith** » Mon Dec 08, 2008 4:33 pm

I just updated my custom DLL function so that its signature has it returning a LONG value instead of a UINT and ensured that Call DLL Function has the return type set to "long."

I have the DLL function returning the value 0, as I've verified with debug logging built into the DLL. However, after calling the DLL function from IA, the variable that is supposed to receive the result is empty.

jsmith · Post by **jsmith** » Tue Dec 09, 2008 11:54 am

Solved it - one of the parameters I was passing in to the DLL function was not defined before calling the DLL function. I looked at the value of all the input parameters in a debugger within the DLL and saw one of them to be "$MYPARAM$". My DLL ran fine, because this is a valid string. Making sure I set it with "Set Variable MYPARAM to" before calling the DLL function made sure that the return value from the DLL was copied to the variable I specified to hold the result.

I don't understand why failing to initialize an input param causes the result not to be copied into the result variable, but it does. Even though this only manifests as a result of my doing something wrong, I'd call this an InstallAware bug. Ideally the MSICode shouldn't compile or should generate a compilation warning if a variable is used without being declared.