I'm glad I found others experiencing the same! I have a 55" Samsung KS8000. I brought the Apple TV 4K back to the store today, it was so intolerable.
While I am no AV expert, I do know how to systematically try everything. I tried two different HDMI inputs, swapped HDMI cables, reset the Apple TV, played with dynamic contrast and smart LED, completely changed my brightness and backlight settings, and couldn't make it look as good and saturated as my old Apple TV 4th generation. The 4K SDR output, manually selected from settings, was quite acceptable, but 4K HDR at 60 Hz or 50 Hz was just unsatisfying. The Netflix and Infuse app icons, which have white backgrounds, were rendering in dull gray. I watched an episode of Bojack Horseman and everything was dull.
Together with the upscaling issues brought out by Nilay Patel and discussed in the main thread, I gave up for now. I will probably repurchase if a future software update claims to have solved these issues, but in the meantime I'll hang onto my $179 + tax.