This is a known problem with 3.0, it happens on some Windows machines.
I think there’s a solution posted for it somewhere in the forums here.
I’ve never had a clear idea of what exactly causes the problem, myself,
though it’s solved in 3.1.0-beta2, because we use a completely different
installation method on Windows starting with 3.1.0.
I encountered the same error when going to the Java website. They have a big green button that lets you check your current version, and clicking that produced the error immediately. I even deleted Java, reinstalled it using the version available at the main website, and went back to the website, and got the same “Cannot create Java virtual machine” error.
I downloaded the latest beta of Java (6u10) and tried it again–no more error. I don’t know if replacing your current Java installation with the latest beta will make a difference with the error you’ve got, but maybe it’s worth a try.