When these instructions fail, you can of course print a message about that. The result however, is that the user sees both your ‘Failed’ message, as well as Joomla!’s ‘Success’ message. Users find this is very confusing.
The solution is actually extemely simple: inside com_install(), you need to return false. This tells the installer that your custom script has failed, and a relevant error message will be displayed. There is an added benefit as well: the installer will perform a rollback, removing all the files in the process.
By the way: I’m blogging live from the Joomlatools Bug Squashing event in Brussels!