Auditioning/recording instruments with no latency?

I have started really enjoying orchestrating by using real-time record-to-click, and layering parts as I go. The results feel much more organic, at least for me. The problem is that NotePerformer has a significant latency. If I use MIDI thru, the delay makes it impossible to record rhythms accurately. I have a nice audio interface and have set the buffer at 32, but there’s still major delay in monitoring (the recording itself is fine, if I strike the key exactly on the click).

I use a digital piano with onboard sounds (A Roland RD-2000), but the quality of the orchestral samples is just not great. I’d prefer to use samples from my libraries. Usually I end up muting the current instrument in NP and just playing the piano sound as I record. But I’d like to use sounds that are more realistic so I can respond to them as I play.

Do any of you have a solution for this? Do you perhaps open Kontakt separately and run samples through that for the current auditioned instrument?