I reinforced the door frame last night. With the depth I left for the 1/2 inch osb and 2 layers of 5/8 drywall the screws for the hinges would have just gone into the frame and the drywall and osb. I removed the OSB and put up 2 - 3/4 inch boards behind it to screw into. It should give a solid backing for the hinges but time will tell if it will be strong enough to support the weight of the door. I'll have to use less drywall on that side of the door to make it work with the additional thickness but I think i have a way to make it work.
(Edit - The drywall is up and to make it work I ended up using only a layer of 1/4 drywall on the side with the hinges. So there is practically no drywall there at all. This gave a substantial frame to screw the hinges into and I was able to put the wood directly next to the frame which gives the frame stability. For the area around the rest of the door I used a layer of 5/8 and a layer of 1/2. This was the only combination I could find that would not stick out from the frame. That being said it fits perfectly flush around the door. Worked out perfectly. )