Hello mmoreo,
There is a function in the mix console that may help you narrow down what is taking some of your latency. Only works on channel inserts so it will not help with any heavy VSTi’s you may have in your project. On the top left of the mix console click the “Setup Window Layout” icon, then check “Channel Latency”. You’ll get a little window above each channel that will tell you how many milliseconds are being taken, you can click on that window to show a more detailed view. See attached pictures, shows my main output channel with the Brickwall Limiter inserted taking 1.0 ms.
Hope this helps,
x.