Help with MIDI Input latency / MIDI output is fine

I recently started using Cubase Artist on Windows 7 x64 as my main sequencer for my external gear. I got all of my MIDI gear and audio interfaces setup with it and mostly everything is working fine. I have two MIDI keybaords that I am using as controllers. I am only using one VST instrument (HALion 4). I am using two USB audio interfaces, a Lexicon Lambda for VST output to mixer and a Lexicon I-Onix u42s for audio input and MIDI I/O. My problem is that I am experiencing intermittent MIDI INPUT latency when playing my synths through either controller while a sequence is playing. MIDI OUTPUT is working fine, so playback is not affected. My notes are being recorded slightly late. It doesn’t do this all of time, but it is doing it frequently. Input timing is fine when the sequencer is not running, every time. I checked my computer CPU and memory performance inside the task manager, and it is all running low (I have a quad core with 4GB memory) when I experience this. I am using the MIDI input on my Lexicon I-onix u42s interface and ASIO4ALL v2 for the ASIO driver. What do I need to tweak to get better MIDI INPUT performance?