I don’t have any experience with the Yamaha, it’s required OS versions, or the stability of their drivers so I can’t comment. You said you had the issues with no third party drivers installed, good to know.
I don’t have any experience with that age of MacPro, ours are 2011 and 2012 models. We use OS10.8.5 with Nu5.5.x. You also have a much older graphics card which may or may not be causing problems. You can only run Nuendo in 32 bit mode on that Mac if I remember correctly, with large sessions you will run out of memory because Nuendo 5 caches high res waveform data in memory during record and it will crash. Speaking from experience…
If your sessions are large I highly recommend moving to a Mac and OS that can run 64bit due to Nuendo’s inability to NOT chew up large quantities of ram with waveform data. If your sessions are small then keep an eye on your memory use.
I don’t know what your priorities are, if the ShuttlePro drivers are not installed I wouldn’t worry about that working - there could be some incompatibilities. I personally would worry about getting the MacPro working first. You mentioned that in Activity Monitor you were seeing 10-15% when firing up Nuendo. Because Activity Monitor displays usage oddly I have to ask - is that 10-15% in the Nuendo row of Activity Monitor, or at the bottom under “User”? If it’s in the Nuendo row then it’s 10-15% of the total cpu percentage available unity-normalized to one core, which on your dual-quad machine is 800%. Apple Math.
I don’t remember what it is when we fire up Nuendo but if it’s 10-15% of 800% then you’re just fine at least to start. Your troubles are elsewhere. You mentioned “jumps in the recording” - that’s a significant problem. If the Activity Monitor percentages are as I mentioned (800% total) is it possible your ShuttlePro is causing the problems? If it’s activated during recording it can cause the timeline to attempt to jump to another location if the driver thinks something is happening.
So, I stated some stuff that may or may not be accurate based on my interpretation of your comments.