Too much heat can shorten the life of electronic components, and yes, with those temps, I'd say things are a little warm. If the cabinet is pretty much sealed when the door is closed, there's no cool air coming in, and the hot air just builds up and is recycled through the case, making it warmer still. Add the heat generated by the other components, and with it rising to the top of the cabinet, it's not a good, long term solution. You may well not have any obvious problems, but I wouldn't be happy if the case is too hot to touch.
How many fans are in the case? You should have at least one bringing cool air in, and another venting warm air out. Your either going to have to remove the back panel, or at least drill some ventiation holes in it. You could always cut out a section and add a grill, which will at least allow the hot air to escape.