If you have a Panamorph UH480 or DC1, both are fully capable of 4K performance. In fact, the DC1 is the only anamorphic lens specifically recommended by Sony for their 4K projectors.
Be curious to know which Panamorph lens you have - they make 4 different models. I would also be curious to know what your throw distance and ratio are. The reason I ask is that you say the image is sharper and brighter without the lens. You should not see a decrease in sharpness with the lens unless you are outside the recommended throw distance.
The brightness issue is very unusual - you should definitely see a brightness increase with the lens in place, at least compared to zooming. The only way the image with the lens should appear dimmer than without it is if you are comparing the 16:9 image brightness to the 2.40:1 image brightness. Of course, the 2.40:1 image brightness will be lower because you are asking the projector to fill 33% additional screen area.
As mentioned, the Sony has lens memory so a lens is not strictly needed, however, using the lens will provide a brighter image plus prevent the black bars from actually being projected onto the wall.