Sync Issue.

Could anybody explain why the following is happening?

I have a Project with an RME UFX Buffer size of 128 samples, containing several Internal VSTi Midi tracks and one External Midi Track - triggering V-Drums live.

To bring the audio from the V-Drums into sync with the Project I apply a -56ms track delay to the Midi part (I find this whole delay thing very iffy, but I suppose it is part and parcel of recording these days).

So, now I have the drums cooking with the other instruments.

If I then increase the buffer size, the V-drums play back out of time (increasing the buffer to, say, 2048 makes the drums play wildly out of time.

Why would this be and is there a simple solution?

It is such a pain to have to re-align all the tracks just because the buffer size has changed.

PS. I know that once I bounce all the midi parts to audio the timing issues should not occur (I hope so anyway).