InstallFinalize: Do you write to HKCU or HKLM? Anything sequenced after InstallFinalize will not run elevated in a managed environment and will hence fail during installation if you try to write to HKLM(unless you change the ACL for the key you write to - or you kick the MSI off from a cmd.exe that is elevated already - not recommended, the package is flawed).
MaintenanceMode: That condition looks like it is Installshield-specific: NOT Installed OR MaintenanceMode="Modify". I think Installshield itself sets that MaintenanceMode property in a proprietary fashion. Hence it is not available in WiX at all.
Testing Conditions: Conditions can be hard to test and you need to do real-world testing in many installation modes (install, uninstall, repair, modify, patch, major upgrade, self-repair, etc...) to be sure they fit your particular case and work as expected. A typical scenario where people are surprised at conditioning is major upgrades - the uninstall of your old version and the installation of the new version - it tends to reveal unexpected problems with conditions in my experience. Often due to the complex sequencing issues and the fact that during such an upgrade both the old setup's uninstall sequence and the new setup's install sequence run. Hence erronously conditioned custom actions can run several times during the major upgrade process and cause a real mess. Please read this answer for more: Run Wix Custom action only during uninstall and not during Major upgrade and this short version.
Testing Tip: To implement a quick major upgrade you need to change the product code and bump up one of the first 3 digits of the version. And you need to have the MajorUpgrade element in there. You can set the product code to auto-generate by setting it to * (I prefer to change it manually). Compile an MSI, suffix it with "_Version1" then do the updates mentioned and compile "_Version2". Run upgrade sequence. You should also point to the updated files for version 2 by changing the source path, but you can roll without it to test conditions.
Modify: For your case, there is a cheat sheet from the Installshield guys suggesting the following condition in their "MSI Condition Cheat Sheet":
Installed AND NOT REINSTALL AND NOT REMOVE~="ALL"- The above condition should enable the custom action to run in modify operations only. If you change it a little it should work for install and modify only:
NOT Installed OR ((Installed AND NOT REINSTALL) AND NOT REMOVE~="ALL")
I will have to run an extra check on that last condition tomorrow, but I just post this so you can get testing yourself. I took out PATCH, but maybe I should add it back to cover major upgrade patches. Will you be delivering patches?
Quick Mockup for Light-Weight Condition Testing:
The VBScript:
Note! Make sure the VBScript file is in UTF8 or ANSI. Unicode will not work. Maybe create it in Notepad rather than Visual Studio. I have seen issues when creating the VBScript file as a WiX text file that I then rename to *.vbs format. Don't do that.
MsgBox "I run, therefore I am conditioned and sequenced"
WiX Markup, Custom Action:
<Binary Id='SayHelloMsgBox.vbs' SourceFile='SayHelloMsgBox.vbs' />
<CustomAction Id='SayHelloMsgBox.vbs' VBScriptCall='' BinaryKey='SayHelloMsgBox.vbs' Execute='immediate' Return='ignore' />
WiX Markup, Sequencing & Conditioning:
<InstallExecuteSequence>
<Custom Action='SayHelloMsgBox.vbs' After='InstallInitialize'>NOT Installed OR ((Installed AND NOT REINSTALL) AND NOT REMOVE~="ALL")</Custom>
</InstallExecuteSequence>
Writing to Log: You can write to the MSI log from VBScript as described in this article: MSI Tip: Writing to the Log File from a Custom Action.
Update: You can eliminate a separate VBScript file for the condition debugging using this approach.
