Thanks.
I think it’s not power related, as I tried running the unit with external power and it made no difference. The latency never exceeds 680us. So that should be alright i think.
I hope the usb card will fix the issue, otherwise I have 48 days to fix the problem before I have to return it.