What you are describing (I think) is a difference between Relative and Absolute calibration.
(As you say, that is not the same as having random calibration throughout the colour gamut and/or grey scale).
The human eye will indeed 'manage' consistent colour errors, effectively balancing them out relative to other colours.
A good example is when grading films (I have graded a fair few...).
As a colourist you get used to the need to keep referencing back to a 'pure white/grey scale' while grading to prevent you perception of colour from drifting during a grading session.
If you don't do that, after 30 minutes of grading you can easily be making wrong colour assumptions.
While you don't 'see' the errors at the time, walking out the grading room (for a coffee, etc) coming back into the room and looking at the exact same shot you were just grading can be a real shock!
So, we use perfect grey scale/white balance references to keep our eyes 'in calibration'.
I used a small light-box, with very accurate grey scale transparencies.
But, back to the original question - while such Relative errors can be 'assimilated' by the human eye, the issue is the Absolute colour error remains, and that means you are no longer seeing the colour as the DoP and Colourists intended, and that can alter your emotional perception of the images you are viewing.
Having a 'cooler' image generates a very different emotional response than a 'warmer' image, for example.
That makes accurate calibration rather important to me