Discovered Monitoring/Latency Issue

Using very fast rig with RME Raydat at 32 samples. I have a few higher latency plugins (seventh heaven, etc.) on my busses, but use constrain delay to turn them off. Only want to have a channel strip/gate working on drums while I record. Issue is as follows: when I only monitor one stereo track (say overheads) with constrain delay on, it sounds fine with very little delay (as it should be). When I attempt to monitor close drum mics as well (also with only channel strip activated b/c other plugins have been turned off by constrain delay button), I get huge latency. It is the same latency I get when I turn off constrain delay (probably the 800 samples for the slowest plug-in on the chain - but those plug-ins should be off when I’m using constrain delay). In other words, when I monitor multiple inputs, the benefits of constraining delay go away even though those other channels do not have high latency plugins activated.

What gives? Monitoring the second mic shouldn’t affect the constrain delay button, yet I am getting completely unusable latency despite having a 32 sample buffer/asio guard off, etc. Ever since Cubase 6 or so, I’ve had issues with latency.

Constrain Delay compensation does not affect the Strip.