Inconsistent video playback

I’m trying to mix audio for a video in Cubase 9.5. I imported the video at the right framerate, and aside from a laggy playback (Cubase seems to need some time for the video to catch up with the playback), I’m noticing inconsistent video playback. For some effects I’m editing on exact timing, and all the sound I edited in one session, doesn’t fit anymore when I play the video once again. I didn’t accidentally move anything or hit a quantize function.

Anyone else noticing differences in timing when playing a video with audio? Any tips on how to fix this?