my problem ended up being that the .exe name was not getting passed properly to the Firewall Exception installer. fixed that by setting the CustomActionData, and now it works. by remoting into the WHS, i was able to manually install the .msi and it setup the firewall exception. still had to manually start the service, but it ultimately showed up on the 360!
the next step is to write a WHS console AddIn. the console AddIn should let the user start/stop the UPnP service and change settings like choosing folder shares. then i'll have to figure out how to get that installed as well :(