Randomly Occurring Latency

This happens approx once every dozen times, doesn’t matter the project. Even with the lowest buffer settings, I’ll suddenly hear 15-20ms of latency. I’ve rebooted with nothing but Cubase, even wifi off. Moved the project around, changed names, recorded to different drives etc. Out of nowhere the buffer settings will seem to not affect the latency I’m hearing.

Yes I’ve searched the forums already for a solution. What am I missing here?