You may have tried to do too many pieces at once OR clamped too hard. It also helps to use the top/bottom/sides as clamping/squaring support when gluing. Basically build the full cabinet each time, even though you're only gluing a piece or two, just to make sure the cabinet stays square. I learned that the hard way on one of my first builds and had to pry pieces apart, sand and re-glue. The ply is more forgiving than MDF if you want to separate them and do that.
As Chris said. Use 2x4 wood across on the top and bottom of the cabinet to apply equal pressure so you can really clamp hard with 3-4' clamps to see if you can square it a bit when you go to clamp the baffle on.