I'm thinking you will never find a catch-all, one-stop solution to this problem. Here's why.
Every console since the dawn of time has worked with overscan, for various reasons. Dreamcast included. Ultimately though, overscan will be used more with older hardware to provide a full screen area, because the framebuffer of many games tends to be on the small side. TV to TV, overscan can vary, but you will find that if you can get the H+V alignment where you want it to get the most coverage with one broadcast on the same TV, every other broadcast tends to keep within that range.
The VGA Box is the wild card. It provides a high quality, non-interlaced (i.e. progressive) image. However, VGA is not a broadcast standard in the way that NTSC, PAL etc. are. It has no means to deal with overscan. So the only option is to turn it off. When using the VGA Box with your Dreamcast, the image that is given has no overscan at all, and it is frequently not providing the exact ratio that was originally intended--that only worked when stretching happened to account for overscan. This was not a problem in 1998 when the Dreamcast was designed. It was only ever presumed that you would have good access to the kind of H+V adjustments on your VGA monitor to correct the aspect and center as required. With your VGA Box now eliminating overscan, your transcoder altering the signal type for your HDTV, and your HDTV coming back to it looking for overscan, things have all manner of propensity to be wacked. At any rate, bad geometry is just bad geometry, you can be sure your box is not generating it. I'd expect geometry from other sources to be bad at 1080i if your Dreamcast through the VGA Box and transcoder box are showing bad geometry. Work with some test patterns, or other sources+modes with/without the transcoder box to confirm this, and go from there.