JMX needs an RMI registry operating on an open port. By default the RMI registry port is chosen randomly at the startup time and it doesn't play well with firewalls.
Since JDK7u4 you can use
-Dcom.sun.management.jmxremote.rmi.port=<port>
to set the RMI port to be used. Then you can enable that port in the security group.
Note the .rmi.
part of the above setting because this usually gets confused with the com.sun.management.jmxremote.port
setting. You should not!
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…