Simon is a hands-on software architect and a senior consultant at 
Securing Pebble #3 : Use the Java 2 security model
So, while disabling JSP compilation and not running as root won't ensure that you have a secure webapp, using the Java 2 security model will help to secure it up from many types of attacks. By default, servers like Tomcat are started up in an insecure mode with a default security manager that allows pretty much anything. Ever tried running a System.exit(0) from a JSP? Try it! Of course this isn't the only problem - you could be able to read/write files, execute shell commands and install untrusted code.
To solve these problems, you can start up your web/app server with a security policy that explicitly states what can be done. With Tomcat, starting it up with the -security flag with install the security manager based upon the permissions granted in the $TOMCAT_HOME/conf/catalina.policy file. Speaking of Tomcat, there's a really useful security guide that describes this in more detail. I can pull apart the policy file I have and post exactly what is needed for Pebble (I have other apps in there too) if that's useful.
Securing Pebble #2 : Don't run your server as root
This is another often-overlooked point, but don't run your production app/web server as root. Should somebody manage to hack in (by uploading a JSP or otherwise executing some code on the server), that code will have access to everything root does. Run your server as an unprivileged user and don't forget to lock down the files they need to read/write.
