I think if I had to choose, I'd rather set color at the correct level even if it resulted in clipping above reference levels. Obviously I'd prefer not to have to make the choice. But in general getting the color levels matched is important to the picture. Lowering the Color setting below the correct level reduces color saturation, which is a big part of perceived quality.
Also, the point of reference levels is that a watchable picture should result if a display clips above reference. Unfortunately that's not guaranteed to work, because the people mastering the video don't actually do the quality control on a display that clips. They really don't know exactly what a clipped display looks like. So it's supposed to look fine, but no one is checking if it really does. But in practice, watching on a clipped display will mostly look OK.
One way to look at it is that having the Color set too low affects the overall color saturation of the display all the time in a visible way (except in black-and-white movies), whereas clipping affects only the brightest colors and highlights.
On the other hand, really bad clipping can make a scene look awful, whereas turning Color down a small amount will generally be unnoticeable to the naked eye. So the worst-case performance of a clipped picture is worse than the worst-case of a desaturated picture.
So I suppose it comes down to how you weight the worst case vs. the common case. Avoiding clipping mitigates the worst cases. Avoiding desaturation optimizes for the common case.
I think I just argued both sides. Does that help at all?