Different buffers for recorded data and incoming data

I think anything above 6ms is getting noticeable, but workable uptil 10ms.
Try setting your buffers at 128 and use methods to bring down the ASIO load like:
Freeze tracks
Constrain Delay Compensation.
Remove any plugins on the Master bus.