Data Execution Prevention DEP Server 2008 Kernel32.dll

[tj]

Does anyone know how to deal with DEP on Windows Server 2008? I am using the kernel32.dll to copy a file that is needed before Apply Install. DEP is catching this (or so it seems) and stopping the install.

[tj]

Looks like I was able to get around this by using RUN Program with a copy command.

HOWEVER, I'm interested to know why the call to the dll is flagging DEP. I actually use this dll in a couple of places and I went that route because it was recommended in a number of places on the forum.

Any insight on this? Is the dll being called in a non-standard way?

[tj]

If someone is interested in tracking this down, here is a tiny app to reproduce. You'll need to copy a sample.txt file to your supportdir.

I am running a Windows Server 2008 Standard server on a VM as a test platform.

Code: Select all

~InstallAware Clipboard Data~
~Terminate Install~
~{927721BC-79EF-43C4-BA7F-6FD14EA93F54}~
~MessageBox~
~{D522582A-42A6-406C-9219-98F9D84C87FD}~
~Made it here~
~Passed the DLL?~
~0~
~1~
~~
~Call DLL Function~
~{D48153F8-2C88-4B14-9916-3EFD8ED1583A}~
~kernel32.dll,CopyFileA,void,LASTERROR,"pointer to string",$SUPPORTDIR$\\sample.txt,"pointer to string",$EXEDIR$\\sample.txt,$~
~mIDEFunc.dll\\mEXEFunc.dll~
~Call DLL Function~
~{C1F29A8A-1838-4C88-A989-D76379764394}~
~kernel32.dll,CopyFileA,void,LASTERROR,"pointer to string",$SUPPORTDIR$\\GideonProtectedConfigurationProvider.dll,"pointer to string",$EXEDIR$\\GideonProtectedConfigurationProvider.dll,$~
~mIDEFunc.dll\\mEXEFunc.dll~
~Run Program~
~{F0359AAF-AE6B-430F-8C39-E9311AAD8323}~
~copy~
~FALSE~
~ "$SUPPORTDIR$\\GideonProtectedConfigurationProvider.dll" "$EXEDIR$\\GideonProtectedConfigurationProvider.dll"~
~FALSE~
~~
~WriteLog~
~{F640F02F-7501-4F51-A999-EA3E0C29B48B}~
~c:\\test.log^|^Copying Provider to EXEDIR: $EXEDIR$^|^1~
~mIDELog.dll\\mEXELog.dll~
~Set Variable~
~{418C088F-4753-4B37-A099-DBB70E527656}~
~LASTERROR$MYAH$MYAH$FALSE~
~~

[MichaelNesmith]

The truth is, it'll probably be flagged on all platforms when DEP support is extended to all applications (and not just the OS). This breaks a lot of applications and I'm not surprised it interferes with Call DLL Function, a low level procedure. Server OS's have DEP extended to all applications by default, whereas workstations like Vista and XP have it pre-configured for only the operating system. This is a switch exposed in the OS settings so you can change it there. There's probably also a registry correlate, so you could test for this, make necessary changes, reboot, and try again all from your setup script.

[tj]

The COPY command I pasted above was actually failing.

I ended up creating a COPY.bat as:
COPY %1 %2

I added that to my $SUPPORTDIR$ and changed the call from COPY to $SUPPORTDIR$COPY.bat I left the parameters in place to pass the file source and destination as parameters to the above batch file.

That actually copied the file and still passed the DEP.

I find it EXTREMELY odd that DEP catches a call to the Windows API copy command but it allows a batch file copy. If the whole point is to prevent malicious activity, I would think that a hacker would use a batch file way before the Windows API command. But, I'm sure Microsoft is smarter than I am...