I think the issue is pretty straightforward...just not to me!
Let's say you've done the calibration, and you get a 100 ms delay. What does that mean, exactly? That your video is 100 ms behind the audio? That would make sense. I'm guessing the game compensates for this lag by delaying the audio by 100 ms...in other words, it makes the audio come out of your speakers 100 ms later. (To sync up with the video, as it's a little behind the audio.)
Now, this works fine in the sections where things have been delayed for you to play in time to the video and audio. However, when you get to the fill sections, where there are no visual cues to keep you in perfect time with your lag, the game is unable to predict which notes you're about to hit, so there is no reference point for the delay....so when you hit the drum, you hear a 100 ms delay in audio and video when you hit a pad. The audio and video are still in sync with each other, but because you're brain doesn't have a built in 100 ms lag, you're now out of time with the audio and video.
Oh dear. I think I need a nap now.
If anyone would care to help clear this up, that'd be awesome!