Latency gradually builds up

I run Cubase Pro 13 on a Mac Studio with UA Apollo interface. I have been doing that for years and it works okay. But recently I have this problem:
When I load the session I have a buffer size on 128:

After some time - 30 - 60 min - a latency is “building” up. Then I change the buffer size to 192 samples and the system is working just fine and I can change back to 128. But after some time the problem is back.
I can’t pinpoint any specific plug-in/instrument just the usual suspects.
What can be done?