thanks for the clarifications, and all the help.
Talked to one of the sales reps I've used at a local home theater store about this, and he got back to me this morning.
He was able to reproduce the issue. All the settings were as people have suggested here, which is how I had them when things were working, but still when connected to the LG TV with the latest update, the blu ray player will not output atmos to the receiver over hdmi 2, only PCM 7.1. He believes that it's because of the way the TV returns it's capabilities now. Technically it SHOULD tell the player that it is capable of stereo / 5.1 / 7.1 / atmos and the formats in which it can deliver those. Apparently on the dolby side of things, it only says 'Stereo / Atmos' (I looked at the TV as an audio source with a PC and those are also the only 2 options listed, which is definitely odd). So when the player's HDMI 1 output polls the TV for capabilities it gets a rather unorthodox response, and adjusts the HDMI 2 signal for whatever reason.
Samsung says that LG Should fix it's handshake response, and LG says that samsung should have made HDMI 2 unrelated to HDMI1. Both are probably true, but as this issue only occurs in this exact hardware combination, and there's already a newer model of the player, I doubt they'll do anything about it.
Today I'm going to see if there's a high quality splitter that will let me send both TV and Receiver data from HDMI1 (Splitter was my first try, and I got atmos to the receiver via HDMI 1, but handshake from receiver said that HDMI was 2.0 and not 2.0a, so the splitter would not send HDR data to the TV, so it might take a better splitter, or a less smart splitter) if that won't work, I'll take the other advice I got here, and get a different player... I was saving up for an oppo, but I might have to find an interim one.
Thanks for all the advice. Sometimes I miss analog.