Huge latency problem (appears after some time in a session)

So I’ve been looking around, didn’t find any fix for my problem and I don’t exactly know what causes it, and I also didn’t always have this problem, it started to appear some months ago.

So let’s say I have a brand new empty project, buffersize at a reasonable size, everything works well, no clipping, no drop outs, no noticeable delay at all (monitoring is really responsive), everything works perfectly. As I add tracks and VST plug-ins, there is no apparent change, up until one point where I don’t know what happens but my monitoring latency suddenly skyrockets and there’s no return from this state. There’s at least half a second delay between my playing and the monitoring. I tried changing buffersize, removing plug-ins, removing tracks… Nothing I’ve tried fixes my project when it enters this state. The only workaround I’ve found is to create a new project and import my tracks, which is really not practical since I can’t monitor the imported tracks. I noticed it often happens when I use NeuralDSP archetype Plini and Abasi, but at the same time, I always use them so I can’t see any correlation.

Anyone with the same problem?

Cubase Pro 11

Focusrite Scarlett 2i2, latest drivers

I have gaming PC components so I don’t think power is an issue (and as I said, it’s a problem that appears suddenly while things work perfectly right before).

It could be as soon as you put plugins on the master bus. As soon as you do this you add the latency of the plugins. Best to leave plugins of the master bus until you have finished recording.