There are a couple of things you can do to fix it. 1) Remove the finish on the front and cut another baffle. Glue, cut correct opening, and refinish. 2) Cut a ring the correct size ID to fit the driver and OD the inside width of the box. Cut it into two equal halves and glue them to the back of the front baffle. Care will need to be taken in clamping in either case. I have used both methods after messing a couple of boxes up in the same manner.
The first one is the easier method if the depth of the enclosure is not set in stone. The second option assumes the current cutout is larger than the OD of the drivers frame making a recessed mounting flange.