Just wanted to update and say thanks to everyone that advised me, in particular HDFury, Dominic, and Claw.
I was very reluctant to open this can of worms since I have a first gen Vertex, and a jvcRS500, and it sounded like it would be a hassle and perhaps not pleasing in the end.
But after watching a few DV titles from the ATV4K, and seeing that my automation that switches between two different profiles/gammas on the JVC still works, and that the custom 2020 Edid triggers the right color space, and that the DV Version improves upon the signal from the ATV for DV titles, it feels even more like I’ll get another couple years out of this projector.
Summary of what worked for me, for others who run across this. (I may have this slightly wrong so I will update and edit as necessary.)
Note that I have an RS500, and have a custom HDR curve set up by Chad that is optimized for HDR10 content with peaks of 1000 nits (or slightly more, maybe up to 1500). With HDR10 content I found that this is most pleasing. I'm using high lamp mode and getting a bit more than 100 nits on a large screen with full coverage of P3 within a REC2020 container. Since the RS500 has a broken HDR gamma, I'm using the Vertex to also remove HDR metadata.
First, this assumes you already have macros working AND that you know what user mode to invoke on the JVC for HDR content. In my case that is Mode 2.
Second, although it seems counter intuitive, the REC2020 SDR detection can ALSO be used to detect LLDV HDR. So I told the Vertex to trigger my HDR gamma the JVC when a REC2020 SDR signal is detected. (I don't recall why this works.)
Third, I'm using an ATV4K but suspect other LLDV capable devices also respond to seeing an EDID that asks for LLDV in a REC2020 container.
It goes into slot #10
on this screen.
You need to use a custom EDID if you want to use BT.2020 for LLDV (Mark Swift's BT.2020 LLDV EDID) -- which is what worked well for me. This tells the source device to send the color in a 2020 format, which is perfect for most sources and (most important) perfect for most displays, since that is usually how they handle HDR color.
Here is a good step by step about what to do with that file: