Advice for working on big MIDI orchestration sessions

Dorico
1

Hi Everyone!

Just wanted to see if anyone could offer any advice or approaches for tackling large MIDI orchestration jobs in Dorico. In particular, regarding poor performance which (I think…) is due to the (necessarily ) high number of “players” involved when distilling down the composer’s MIDI. I’m running a 2010 Mac Pro with dual quad core Xeons and 64gb of ram. Dorico is running of an SSD and the playback template is set to silence. Not the latest and greatest, and I’m not opposed to planning an upgrade, but the performance is pretty frustrating when I have to work on the large orchestra MIDI sessions, especially when compared to my other programs (which I obviously detest for various other reasons…)

I clean the MIDI first in DP and then import a SMF into a template.

My old method in ‘the other programs’ was to bring in the cleaned midi to a dump file and copy the notes into the finished template file. Unfortunately there is a horrible lag when switching between two open files using Dorico, which is kind of prohibitive to this approach after a while.

I think all the constant layout calculations are what is slowing it down right? It seems utterly primitive but I’d love to turn them off while I’m in galley view and just moving the notes onto the right staves… Wondering what you all do to handle this kind of task in Dorico? It’s downright wonderful when it comes to producing the finished engravings but I’ve just gotten so frustrated with the performance on big jobs that it’s really starting to do my head in!

Looking forward to any solutions there may be.

1 Like
2

Hi mate,

I have a similar situation. Running a 2017 MacBook Pro (with top specs) and an Ultrawide monitor.

I’m considering buying a new computer to solely run samples off (through VEP). I think that this will go a long way to solving my problem… but I’m still researching this.

The other thing I’ve done is duplicated plug-ins, and also purged samples once I’ve finished an instrument.

That’s all I’ve got for ya!

3

Hey thanks for your advice!

I actually think we are talking about almost opposite sides of the coin! I haven’t got any samples loaded at all and my playback template is set to silence - it’s just a lot of midi that I am orchestrating down to a regular orchestral score for recording purposes. I wish this was slick and smooth but it’s just like chugging through mud. It is only in the realm of 1-5 seconds but it really adds up and breaks my flow - at least it encourages me to be very precise because any mistake is extra annoying…

4

I guess that’s what I get for trying to reply at work haha, sorry for misreading!

You should upload the project and a diagnostics report so someone can check it out thoroughly. Does THAT help? Lol

5

What do you define as a “large” project in this context? Number of players, length of piece?
Do you have condensing activated in some layout? Disabling it while editing can give you a big performance boost.

6

Hey!

Thanks @Estigy! Your question made me realise something - I don’t consider this a big or large score. It’s actually just a normal session it’s just that Dorico seems to run a lot better when the session is smaller: like if I’m just doing a strings orchestration or something.

The types of scores that (for me) Dorico has a hard time handling are when I have a whole orchestra MIDI sequence to put into a finished score to record. So I might have 30 staves for the finished orchestra score plus between 30-80 midi tracks from the cleaned up MIDI sequence. I try and reduce as much as I can as I’m doing the cleanup but in some cases I actually create new midi tracks where a composer might have written disparate voices on a sustain patch, for example.

I don’t have condensing on. I’ve cleared out all the unnecessary layouts and only have the single full score. I have disconnected and disabled (in Dorico prefs) all my MIDI input devices at the moment. The program itself feels great, like moving around and everything is completely seamless. It’s just everytime I do pretty much anything it takes about 5 seconds! I have my doubts that it’s anything to do with my machine in the sense that I’ve done many scores in Sibelius of this scale without any of this lag . Not that I enjoyed it, for many other reasons, and not that I want to suggest that I should keep my old workflow. Far from it in fact! I would simply like to know what the recommended approach for this common (at least in my line of work) situation - because it sure feels like I am doing something wrong!

Thanks everyone.

7

From Dorico’s point of view, a project with 80-120 staves is certainly large. The problem won’t be RAM or disk speed, I’m afraid, but purely CPU grunt and Dorico’s computational efficiency. Even though Dorico will be splitting things up as best it can to make best use of the available CPU cores in your computer, because it has to marshall all the results together (in the middle of processing, at the point at which it needs to space the music, and at the end of processing, when it produces the final result), there are still practical limits on how much parallelisation can be performed.

If you have the option to set up temporary custom score layouts that contain, say, the destination string staves and the source string staves from the MIDI data but omit all the others, so that you’re down to, say, 30 staves in that layout rather than 120, that will help, but I realise that might not always be convenient.

In Dorico 4 we are introducing two new tools that are designed specifically to help make these kinds of MIDI orchestration jobs easier. I don’t want to talk too much about them because I would like to retain some element of surprise as to what features are coming, but please keep an eye out for Dorico 4, as I think it should help in a couple of ways.

Dorico 4 will also have improved performance in some areas, but we know there is always more we could do to try to improve its computational efficiency, and Dorico 4 will not be a panacaea in that regard – though I certainly hope you will not find any circumstances in which it is slower than Dorico 3.5.!

11 Likes
8

Hi all!

Just continuing the conversation here as I still find this workflow rather problematic! Importing big midi sessions from composers and producing the final score is about 90% of what I do with Dorico. Here’s my take:

The main problem with the current system is that I think we are wasting a lot of time computing the layout implications of staves that are never intended to be viewed in the final score. For historical and reference reasons it is highly preferable to have the midi stave right there in the same session as the final notation, but it is never desirable to have them in the printed score. Is it possible to have the midi staves be visible only in the galley view? I believe this or something even better would be highly desirable for most orchestrators.

It is rare that I receive midi that really even vaguely correlates to the final staves which does render most of the midi import features (which I’m certain are algorithmically impressive and amazing) to be largely useless. Typically the midi I receive is in the form of “Hi strings long” etc . I have a hard time making orchestration decisions at the midi import stage so I really feel that I do need to import these as ‘temp’ staves in order that I can distribute the music once I can see it. As such I don’t really feel like I can take advantage of the stave saving capabilities of the midi import. Any advice or hope would be much appreciated. I’m beginning to lose faith!

For what it’s worth I use the new Apple Silicon now.

9

Hey fhrw, we’re pretty similar. I too had a 2010 Mac Pro until I finally got a Mac Studio a few months ago haha.

I still stick with a midi dump document, and copy/paste into my actual orchestration document.

You said in your old post that you do cleanup in DP. I find doing my own cleanup super helpful as well. For one, I like to do my midi in batches, rather than taking the whole thing in at once. So, I might export strings as a separate midi file from the brass, for example.

I’ve also found the midi import functions to be VERY helpful with this process too, you just have to practice it a little bit. For example, while I too would often split out multiple voices when I would import to Finale (because if this (screenshot below) were ever imported into Finale, it would be a disaster)

Screen Shot 2023-06-13 at 10.43.11 PM
Screen Shot 2023-06-13 at 10.43.11 PM1254×580 17.7 KB

However, Dorico will automatically recognize these and split them out into two separate voices now. So, when doing cleanup knowing I’m going into Dorico, I won’t separate this sort of thing anymore.

As such, if the midi came as multiple tracks (say, the composer didn’t use key switches, or I split it out for them based on the stems because they did use key switches) so I have one track for tremolo, one for harmonics, one for staccato, one for sustain/legato - If I label my midi regions properly, Dorico will combine them all together and apply the correct articulation markings.

As such, I’ll try to split out, at least roughly, the vln1,2,vla,vc,db material, and label them correctly. When it comes into Dorico, it’s…kind of already orchestrated! 5 staves, articulations switching as needed, voices already split when needed! It’s incredible.

With the Midi Memory function they now have in Dorico 5, if you adopt a consistent naming convention, it will recognize those labels and automatically assign them properly going forward, which is a mind boggling time saver. So now, I make sure I label Vln1 Them, Vln1 Harmonics, Vln1 Stacc - and they all import into a Violin 1 staff, while all the ones labeled with Vln2 go into a Violin 2 staff, all labeled! It cuts down on the number of staves in the dump file dramatically.

Pro tip, I’ve also set up in Midi Memory a setting for string sketches (if I don’t want to do all the split out stuff in midi first) to import as a “violin and cello” pair of staves, as I’ve learned that importing as a grand staff does not copy/paste the way I would expect when I try to bring it over to the real orchestration document.

If the composer uses lots of key switches, I dunno what to do there - the composers I’ve worked for haven’t been big on keyswitching so I haven’t had that headache yet.

10

Also, Instrument Filters might be really helpful for you to cut down on the layout processing. If you make a filter set for each orchestral section, that should cut down on the computing dramatically, I think.

11

Hi @fhrw and @Composerbell … fortuitous that you’ve kicked this off again as I’m absolutely baffled.
I work as an orchestrator as well, dealing with enormous sessions from composer, and the resultant large sized MIDI dumps.

After cleaning up in whichever DAW they delivered in, I have always dumped out a MIDI file so as to preserve every single note from the composer, and then imported this into notation software (previously Sibelius).

I’m trying this now with Dorico 4.3.2 on an M2 Max 32GB, and the load times are, frankly, ridiculous. Importing a MIDI file into the template (same flow) that has 280 tracks … I gave up after waiting 15 mins. I trimmed that down to 114 tracks (the only ones with notes in them), and I got this down to 5’30", which is still utterly unacceptable.

I realise Dorico can rationalise things down on import, but that is far too fraught with danger - I need to see a MIDI version of the sequencer dump, that I can then move around and condense as I go … it is too easy to screw up when the machine does the thinking.

I hope I’m doing something stupid on the import - I’ve tried loads of different settings on import hoping I’ll stumble across what is taking all the CPU/time, but alas …

I love what Dorico can do, but not being able to import MIDI quickly is a total deal breaker.

Any help would be magic!

Cheers and many thanks

1 Like
12

@bodgey The Midi Memory function is one that you train it, so you know when you name things a certain way (like Vln1 Trem) that it will always import that as a tremolo articulation on Violins.

As far as the voice split aspect goes, I went through so much trouble to ensure that the import was correct in Finale, because it would take the tiniest of note overlaps, and import that as an entire whole note chord. While I still check the note ending are correct, Dorico will interpret tiny note overlaps as slurs and mark accordingly -THAT I do have to go through and delete, if I missed any overlap bits. But, a quick “Filter-slurs-delete” and that’s solves in approximately 1 second.

I would suggest to you, as I did to fhrw, to try splitting the process out a little more. Personally, I find a 100 stave tall midi import intimidating, and easy to get lost in, so I prefer to export my midi in smaller batches, which means they also import in smaller batches - which should also help with your load issue without altering your actual import or export process at all.

As with everything else regarding Dorico, I find that you have to learn a new way of doing things to really make it work best for you. Trying to treat Dorico like its Sibelius or Finale is a good way to get stuck trying to make it do things that it’s not built to do, because it’s designed around achieving the desired results via a radically different method.

13

A long shot - would ‘round-tripping’ this via another notation programme prove fruitful.? i.e. import MIDI to xxx other notation app, export a MusicXML file from there, then import that to Dorico.? (not sure if I fully know what I’m talking about here.!)

For that matter, am curious - are other notation apps much better at the importing of (large scale) MIDI data.? I mean, wondering how far off the mark Dorico is with this aspect of workflow.? Are others only slightly better (Sib/Fin/Notion/Musescore/Overture).?

(just asking… don’t know what expectations should be, that’s all…)

14

@fhrw, it’s certainly possible to have instruments that don’t appear in the full score for printing. Probably the simplest way to do it would be to have two layouts: one that includes both the non-printing instruments resulting from the MIDI import and the actual ensemble, and one that includes only the actual ensemble. When you’re doing the orchestration work, you can work in galley view on the layout that contains everything, and when you’re done (or nominally done) with that phase of work, you can switch to working in the other one.

Obviously if you’ve got hundreds of tracks, then it’s going to be pretty unwieldy, so I would suggest making good use of the instrument filters feature to create filters for e.g. the tracks for violins plus the “real” violin instruments, the tracks for horns plus the “real” horns, and so on.

@bodgey, welcome to the forum. It would be great to get a look at one of your MIDI files that’s taking 15 minutes to import – that’s definitely not what I would expect. Creating hundreds of players will certainly take a while, but 15 minutes is a lot longer than I would expect. I imagine you can’t share the file publicly, so please either send it to me via a private message or by emailing it to me at d dot spreadbury at steinberg dot de.

15

@dspreadbury thanks for the welcome, and the quick response.

I’m slowly working through the issues … it seems the issue is importing into the template (same flow). When importing into a blank slate, it’s fine - quite quick. The problem is importing into the template. Unfortunately all the files I’m using are private, and I’m bound by NDAs … if you have the scope to try and recreate the problem your end, I’d be most grateful. The template is a triple winds, large brass, percussion, strings, choir orchestral template. Failing this, I could build a separate template and a mock MIDI file to attempt the recreate the problem and send them.
I’m just not sure what Dorico is doing to create such a bottle neck of work on such high end machines.
One possible workflow seems to be importing the raw MIDI into a blank file, exporting this as a flow, importing the flow into the template, then painstakingly adding the ‘players’ from the template initial flow into the imported flow, then deleting the first flow … bit of a kludge, but workable I guess.

1 Like
16

Thanks for the replies!

I just want to say that I also regularly work on sessions that bring Dorico’s midi import to its knees. Just on a project I have in front of me the average track count I need to import is ~150 so plus the big orchestra I just struggle to see that this program was really designed for this kind of work!

I’ve just been doing some experimenting and have found that opening the midi into a blank file is very quick. That’s only saving on the staves in my full orchestral score though so how come it’s so much faster?

So using the Dorico import algorithms is simply the only way? I just really can’t see myself trusting those. I mean I actually can’t really see them being able to understand most of the midi I get to be quite honest!

1 Like
17

I thought I should explain my confusion over the midi import.

The kinds of composers I receive midi from regularly use many different kinds of the same instrument in the making of a mockup. Looking at the Dorico import window it’s very hard to get a sense what the midi is doing at a given time. This in turn makes it very hard to know where to assign some notes.

How can I tell if “8D oboe” is playing at the same time as “Secret Custom Oboe”? How could I decide just by looking at the import dialogue where to send that midi, or whether it was safe to combine or not (I frequently encounter doubled patches for sonic reasons). What about ensemble patches that contain the essence of an important gesture? I can’t see that when I import, only after.

I can’t see how I can get around needing to import these staves short of just making all those decisions outside of Dorico (also very hard).

But then what is Dorico really for? If I am expected to import my completed orchestration into Dorico for engraving then that’s not really feasible as part of my workflow.

I understand that I can make layouts and what not but the crux of the issue is that I have to import these staves and yet I never want them to appear in any layout at any point.

2 Likes
18

We assume that you typically have a DAW or sequencer available to you and are looking at the MIDI file in the sequencer as well: the orchestrators we spoke to said that a common workflow would be that you receive the original DAW project, or at least a version of it that includes both MIDI tracks and some bounced audio of the samples, either individual stems or a stereo mix. So we are assuming that you have some familiarity with the composer’s template before you start the import, and you’ve already got some idea about what the many tracks for, say, oboe will actually correspond to in the orchestrated score.

The hope is then that once you are receiving multiple cues from the same composer, the same template will typically be in use, which means that the mapping you do for each new cue is just to add the instruments that correspond to tracks that haven’t previously had music on them.

It would really be helpful to have some data, e.g. a Dorico project and a MIDI file to be imported into that project, so that I can see the specific nature of the performance problem you’re experiencing. At the moment I can’t think of a good reason why importing into an existing project should be so much slower than importing into a new one, so we really need to profile it.

19

Hey @dspreadbury and all the other helpful replies!

I realise I must be one of the ‘nightmare customers’ in this instance so thanks for bearing me.

I think I can understand where you might be going - I do have a lot of reservations about changing the composer midi, and not being able to refer to the midi track names to understand the intended sound. But are you proposing that perhaps I should pre-prepare all the midi into a Dorico friendly format to allow for the auto import? Ie. while looking at my daw session I could conceivably condense the various oboe patches down into one or two tracks that I specifically name so that Dorico can remember them.

Regarding composer templates: I do think you are absolutely correct when you say that we typically receive the original DAW project or midi. I think the incorrect assumption is about how formulaic the composer delivery often is. Which is to say - it’s not very formulaic most of the time. Also in the making of a mockup they will mostly be layering or switching between many different oboe patches just to get a nice sound. I think from what you are saying the correct Dorico approach is to compress all of this information upstream so that Dorico can handle the import. The danger is that in so doing there is the chance to lose information and thus make a mistake and thus get sacked.

There’s a coupling between the galley view and the page view that I’m not sure is necessary. Can you see where I’m coming from with that? I imagine the galley view as an infinity zone. The page view could be a function of the information in the galley view. We already have staves in the galley view that are not present at a given moment in the page view (with condensing) so is it really that much of a stretch to, say, be able to flag a set of staves as ‘draft only’ and thus not in the finished product? Do you see what I mean?

I’m not ruling out just doing this whole part of the process in some other package and then just importing to Dorico for the features I can’t live without (condensing, nice engraving etc) but it seems a real missed opportunity to get this really awesome!

20

By the way, I will work at talking to the powers that be about sending you some session data and a Dorico template. I promise you I’m not making this up.

I mean I do wonder why there aren’t more threads like this, and it does give me pause. I can’t be the only one in the world working on this type of material (which is largely pretty normal as far orchestral DAW sessions go). So either I am doing something wrong, or perhaps people just aren’t using Dorico for this purpose?

The dump session method does work okay, but it’d be so fantastic for idiot checking to be able to have all the info right there to refer to at any moment. This is another reason why importing into the finished score is so fraught! There’s too much magic and I can’t easily just look and see which set of notes came from where. Like the music on the oboe stave(I promise I’m actually a trumpet player) should be a function of all the midi oboe tracks. But I’m not necessarily ever able to just throw away the composer midi. How can I verify if I’ve missed anything if all I do is just import the midi via the magic import?

EDIT: I have PM’d you Daniel :slight_smile: