I moved a site with FCKEditor from ColdFusion 8 to ColdFusion 10. When testing the file browser capability I was seeing the following error:
The server didn't reply with a proper XML data. Please check your configuration.
The FCKeditor I was using relies on a ColdFusion connector that calls a custom function called FileUpload. This function is located in /editor/filemanager/connectors/cfm/cf_command.cfm. Since FileUpload() is a reserved function in ColdFusion 9+, the server was throwing a 500 error and the FCKeditor was displaying the less than useful error message above.
The fix was pretty easy. Just do a search and replace in /editor/filemanger/~. I replaced FileUpload with fckFileUpload. You will end up changing nine files total. That's it. I probably should have simply updated the editor, but there were some custom js configurations that I didn't want to track down. So... there you go.