How come I get offset recordings with Direct Monitoring?

Cubase
1

I record with direct monitoring with Lynx AES16 and Aurora16 converters.
All is well until I have long sessions, where all of a sudden the playback start being out of time.
For example, an overdub is played and recorded on time, but when I play it back it is offset.
Usually late by a few ms.

Hope someone has some insight.

2

Sorry for the bump, but I changed the subject to be more clear