I have a Syntax Olevia 537H and I am thinking about getting a mini with a HD tuner (probably the hybrid) to record OTA shows. The 537H plays great 1080i through my 360 and OTA broadcasts.

I noticed that the native PC resolution on the 537H is below the 1080i resolution. Does that mean that the mini will playback video in that lower resolution thereby, lowering the quality of the 1080i broadcasts?

I'm just wondering if the mini will kill the quality of the OTA HD for a 37'' LCD TV.

My other option would be a Tivo Series 3 (I currently own a series 2 and it looks horrible on my HD TV and doesn't recieve OTA HD).

What is a Mini?

I think the Olevia has a 1366x768 resolution so you'll need to check around to see how others got that resolution to work--you want as close to pixel-to-pixel as you can get for the best picture. There's a good primer for ScanResX (http://www.avsforum.com/avs-vb/showthread.php?t=751713). But, you can at least try with a DVI-to-HDMI cable and see what the Mini sees.
As for Live TV, only the newest Intel Minis (http://www.apple.com/macmini/) will play HD a full resolutions, and with the hybrid, you don't get QAM (i.e. Digital Cable), which would be the advantage of the S3 TiVo. And, the Hybrid may need and extension so it doesn't crowd out your USB ports.
Check in the TiVo threads on what are the best resolutions for the Olevia (or even the flatpanel therads too).
Then you need to decide what is best as the combo of a Mini plus tuner is around the same price as the S3 TiVo plus guide fees. If you're upgrading from your Cable Cos DVR it might be a better deal, otherwise OTA is free and you can use the mini for other things.

I tried 1360x720 with DVI to HDMI but that looks bad.

What is a Mini?http://www.apple.com/macmini/