It's probably going to be a communication issue of some kind. Obviously your PS3 sees that it's connected to something 3D-capable, otherwise it wouldn't even enable the feature. But your AVR may not be able to properly recognize that the PS3 is 3D capable, so it's balking at the data it's being sent.
There are some 3D game demos on the PSN store, you could try those. Super Stardust HD is a good one. That will at least tell you if it's the PS3 having a problem with the Blu-ray specifically, or whether the AVR is rejecting all 3D content sent from the PS3.
In the interest of troubleshooting, it might not hurt to temporarily run the PS3 straight to the TV. Try the HDMI port that the AVR is using, and try a different port, see if there's an issue with the port. One of the ports on my old Westinghouse TV didn't work with PS3, but the other one did. Worth checking out. Try all kinds of configurations, and keep track of where the problem goes. Does it follow the PS3, or stay with the AVR? Does it only happen on that one port, or on all of them? Troubleshooting is all about gathering information, and the answer eventually presents itself.
Welcome to Rivendell, Mister Anderson.