6.5 on PC - Midi Controller not working properly

I am running Cubase 6.5 on PC. Windows 7 Pro 64bit. Sometimes my midi controller is not working. Then other times it works using virtual instruments but not with midi. Additionally, when using midi everything will be fine for about 30 seconds then the midi freezes then all the notes come crashing in at once. It’s hard to explain but I know you guys know what I am talking about… it’s like midi gets all dammed up then pushs the info through all at once trying to catch up with the progression of the song which continued rolling. My ASIO buffer is set at 512. Do I need to increase the buffer size to deal with this problem? There doesn’t seem to be any latency from controller to DAWS. Thanks for your help my bros.