What you want does not quite exist at the price point you desire.
However,You can get some used SD encoders for $1k..take the asi out to the drake modulator and upconverter.. and you'll have what you need. But, that will take up and entire 6 mhz QAM..unless you add an ASI muxtiplexer, which usually can't be found cheaply. That scenario will be no more efficient than using a single analog 6mhz. You don't have to worry about PSIP, you may have to do some creative channel setting and encoder program assignments to prevent conflicts with OTA's that you may have on the system.
Also, there are no tuners that I am aware of that will tune ATSC and QAM mixed on the same cable, it's either or. Seems like Drake makes and ATSC modulator card; IIRC, or used to.
Why do you need a "digital" lobby cam? All new Tv's still have analog tuners,
so why not leave that channel analog? That's what I'm doing for the general stuff like security cameras..and using Harmonic and JVC encoders into the Logic Innovations multiplexers and Tandberg,Harmonic and Drake QAM modulators. This is where I put the Studio, weather, and all the HD/SD sources that feed all the offices around the building.
The views expressed here are my own and do not necessarily represent those of ABC, CBS, CW, FOX, MeTv, or AntennaTv; my employer; or its parent company.
Nor my wife for that matter!