Yes, Nuendo, RME, and DM2000 are a fantastic combination. once you get your audio patching resolved and the midi connection through Yamaha Studio Manager correct, when you save the Nuendo project all the settings in the DM2000 are saved with it. Recalling a project lets you restore the DM2000 settings for that project. Meanwhile the DM2000 can be a fantastic, comprehensive hardware controller at the same time, via the remote layers. Great combination.
Sounds like you need to straighten out your routing. At 48k, in groups of 8 channels. DM2000 slot 1 ADAT 1-8, slot 2 ADAT 9-16, slot 3 ADAT 17-24, slot4 ADAT 25-32.
Mic’s pre’s 1-24 direct outs feeds DM2k slots 1-3 (1-8), which feeds rme adat 1-3 (1-8) IN, which feeds Nuendo inputs 1-8. repeat for slots 2, 3, 4
Nuendo channels out feeds rme adat 1 (1-8), which feeds DM2k slot1 (1-8) IN. Repeat for slots 2-3, which feeds Channel 25-48. Channels 25-48 are routed to the Stereo mix to monitor.
Also check that RME Total Mix is correctly routing 1-to-1 all inputs to outputs, outputs to inputs. Set Nuendo Direct monitoring to your liking. RME handles Direct Monitoring seamlessly.
My setup is different. I use 32 outboard mic pre’s into Apogee converters which feeds RME into Nuendo. 32 outputs of Nuendo feeds back into RME which feeds my DM2k. I use DM2k to mix, monitor, and feed headphone mixes to musicians. Yamaha Studio Manager via midi keep the DM2k settings saved with each Nuendo project. Very cool. My Nuendo templates include the DM2k routing and setup. Important when switching sample rates to 96k because the ADAT SMUX setting require the DM2k to use inputs Slot channels 1,3,5,7 instead of 8 channels. You need a 48k template and a separate 96k template to switch DM2k routing between non SMUX and SMUX. Nuendo does this beautifully via Yamaha Studio Manager.