Why do stereo out effects cause midi delay when reopening the same project?

I have had this problem several times before but only today I understood why it was happening.

Yesterday I recorded a normal project and did an audio mixdown using Ozone 9 on my stereo out for dynamics etc.
Today when I opened the project again, I had a huge midi delay coming from all my different keyboards.
There is no audio latency problem as there are only a few tracks in the project.

I read in an old post that someone was using a multiband compressor on the stereo out and by removing the effect, the midi delay problem went away.
So I removed my Ozone 9 and like magic there was no midi delay anymore.

Why is this happening Steinberg?
There must be a way to fix this.

That is audio latency caused by Ozone, the plugin needs time to process and that’s perfectly normal.
There is Constrain Delay Compensation you can engage to make it easier to record by temporarily disabling high latency plugins.

Well, why don’t I have this problem when I am working with the project in the beginning?
I use the exact same number of effects, including Ozone 9, and there is no delay.

This problem comes only when I re-open the exact same project a day later.
Why?