We had a installation on a Windows 2003 Server that failed at the communication with IIS 6.0 every time.  We tried a number of different approaches to remedy the problem but always came back with the same result.  IIS would serve web pages just fine by itself but failed when FileMaker would try to create its link.

FileMaker Server would install successfully and host files using the native sharing over port 5003 without a problem.  But every time it came time to connect with IIS and enable Web Publishing we would receive the error “Web server test failed.  Return code = -1″.  The jakarta redirect would never install but fmi-test would.  Also FileMaker Server would kill IIS every time it tried to link up and IIS would restart.

We called FileMaker Priority Support and they gave some hints but nothing solved the issue.  Here some of the many steps we took:

• Removed and reinstalled IIS, Manually removed fmi-test
• Removed and reinstalled FMS, removed FileMaker Server folder
• Removed all components – IIS 6.0 , FileMaker Server, Java, Apple Bonjour
• Change Port to 591, Disabled Filewall connections

Suggested Steps from FileMaker Pro Support

• Remove and Reinstall FMS and IIS  (It’s their favorite step!)
• Change Ports
• Installed with network card disabled
• Reboot Server with only Microsoft services enabled and then manually start Apple Bonjour and FileMaker Server and try the deployment again

None of these worked and It was on a production box that could not be easily rebuilt as it was doing other important tasks as well.  Another good reason that it is always a good idea to have a dedicated machine for your Production databases.

In comparing installations to another Windows 2003 Server with FileMaker Server installed and PHP/IWP working, we finally found that an old employee had change the directory rights on the FileMaker root folder.  He had removed the Users group and explicitly gave himself access to the directory, sub-directories and files .  As soon as we restored the default permissions as they were supposed to be to the Filemaker folder everything worked like magic.

Lessons Learned

• DO NOT let non-server admins mess with your production servers even if they think they know what they are doing
• When re-installing FileMaker Server remove the root FileMaker folder and not just the FileMaker Server folder.  (Don’t forget to backup your data.)
• Continue to encourage clients to have a dedicated FileMaker Pro Server