I have been working on a orchestral score. I have had several flows as experiments I tried the condensing score for fun, but turned that off. I had several custom scores for WW, brass, strings, etc. Eventually the Dorico file got so big and slow that it became impossible to use. It takes over 3 minutes to load on a pretty fast Windows computer. Worse, just about anything you try to do puts you into a 3 minute delay.
I followed the advice on several of these threads and none of that worked. I applied the “silent” template, but the file remained huge (about 26 MB). I eventually decided to create a new score from scratch. It literally took me 3 hours to cut and past the music to the new template. I did it one section at a time (woodwinds, then brass etc), then removed the players and layouts from the problem file and saved both files. Each of those cycles took about 30 minutes because it involved a half dozen coffee breaks for the 5-minute delay.
In the end, I think my new template file is OK and not causing a performance problem yet. But I eventually deleted all flows and all but one of the players from the problem file, leaving a single measure with no notes. And it is STILL almost 26MB. And it still takes 5 minutes to load. When I examine the Dorico file, I find supplementary_data\vstaudioengine\ accounts for almost all of the file size, even though I have no VSTs in the project at all (and no notes).
I bet many of the problems people have reported on large scores are some variation of my problem. I hope somebody will take a hard look at this file. If you can solve this, you probably will eliminate most of the large-score performance complaints.
I found that by using 7-ZIP, I could actually delete the VSTAudioengine data. The file still opens in Dorico OK. You have to go into PLAY and apply the HSSE template then restart Dorico in order to get the playback to work, but that reduced the Dorico file from 26 MB to 5MB and the performance was much more acceptable. I fear that this is a creeping problem that will work its way back up to the impossibly slow level.
I have applied the above technique to the score I had been working on before performance became impossible. I am undecided whether to resume with that file or to switch over to the newly cut/pasted file I built from the Dorico template. The template file lacks many of the options I prefer, but might be safer.
Are there any suggestions for making this vstaudioengine data much smaller?
Why didn’t it go to (basically) zero when I applied the silence template, which removes all VSTs?
Is there an easy way to load lightweight general midi voices, at least during the early stages of the project when editing speed is much more important than rich playback?
I notice your signature says you are still using Dorico 2, which means you cannot yet take advantages of any optimization made in more recent versions.
I’m guessing he’s actually using Dorico 3. When I tried to open his file in Dorico 2 I got the “project created in a later version” message. In Dorico 3 the 26 MB file initially appeared to open instantly, but then switching to Write mode took about 3 minutes. I deleted the Soprano, added a Piano, and when I saved it the resultant file was only 399 KB! The new file seems just as fast as any other. Very strange.
Actually, simply opening the 26 MB file, waiting 3 mins or so for it to be usable, and then saving it under a different name without doing anything else results in a 397 K file.
It appears that there could be some kind of data corruption in the audio engine data, although it’s difficult to tell without seeing the original project. I certainly have never seen such a huge audio engine data file before. All the time was spent waiting for that file to be opened. What plugins were you using, and how many? Dorico just asks each plugin to save its state, so you are totally at the mercy of plugins how much state they save. Normally for HSO I would expect about 3Mb per plugin in the rack.
The trick to delete the audio engine data chunk does make it possible to open quickly (only took 2s once I’d deleted it). There was another warning internally about a bit of duplicate data in the Dorico part of the file, and I’ve done a specific fix for that which will be in the next version (which can also cause projects to be very slow to load). If you have this problem on another project then try backing it up, deleting the engine data file and then applying the playback template again.
But this 3-minute delay is not just at open time. It seems to happen quite often. It had gotten to the point that it was completely impossible to use the software. I either had to figure this out or switch this project over to Finale.
The only VST I had beyond the 3 instances of HSSE that were created automatically was an instance of Rapture Pro I use to voice the chord playback. However, in all these examples, I had already applied the silent template and had ZERO VSTs showing, yet file size still of 26 MB.
I am thinking this is not so abnormal on a full symphonic score. The symptom is that during the unresponsive 3 minutes, one of the hyperthreads is pegged continuously. Even though my PC shows only about 20% CPU usage, basically the entire system is locked, even unrelated applications. I notice from 7-Zip that the VST data is compressed about 10:1, so that means my file was carrying around something like 250 MB of data. My theory is that the VSTs are deciding they need to decompress that whole stream much more frequently than you might expect – not just at load time. And decompressing 250 MB is not a small task. I should note that I have 32GB of RAM, so it would probably be better if the uncompressed data were kept in RAM and referred to directly rather than having to process the entire 250 MB each time something has to be fetched.
There might also be some interaction with anti-virus software. If the AV program is also deciding to scan that 250MB stream, that will slow things down. In my case, I use Avast, and I didn’t see Avast using much CPU during the unresponsive window.
This is really peculiar because I saves this file in multiple stages as I was systematically removing players and measures. And during the entire process, I had NO VSTs CONFIGURED.
What I may not have done is open the file, then immediately save it to the same name without doing anything else. All of my saves were to new file names so that I could preserve the evidence trail.
=======
On edit:
Fred, I suspect the vstdata was eliminated when you saved the project because you don’t have the Youlean VST on your system. And that is why the data remained in my Dorico file even after multiple saves.
The file I uploaded above was the result of 5 successive steps where I kept deleting players, layouts, and measures, trying to get to the point that something might behave positively. As I said above, I had already run the SILENCE template. There were no VSTs at all in any of those 5 stages. Yet I made it through 5 saves with the file remaining massive.
It was not until later that I realized I could look inside the .Dorico file using 7-Zip. That’s when I realized that 99.9% of my final file was VST data – even though there were no VSTs. And this being compressed data, that represented about 250 MB of “VST data” that Dorico is evidently trying to read quite frequently. That cannot possibly be right. Even if the wasted data were smaller, it would still have some impact, so IMHO it behooves everybody to figure out what is happening here.
There have been quite a few complaints about intolerable levels of performance when working on large (i.e. many players) scores. I bet most of those complaints are variations on this situation.
Evidently I have a workaround by using 7-ZIP to purge the bad data and then re-run the HSSE template to reload the VSTs with default information. But even that becomes 6 MB or more compressed (i.e. 60 MB of raw data) being carried around n the file and possibly being read more frequently than necessary. Time will tell if this continues to grow back to the 26MB level.
It looks to me that the problem is the Youlean loudness meter plugin, which is storing the massive 229Mb plugin chunk. I guess that’s an insert in one of the channels. Applying the playback template won’t (currently) remove any insert plugins. So the first thing I would suggest is to unload that and re-save your original file. If the plugin isn’t present then the file should shrink massively. We have never had any previous reports about this massive increase in file size so it may suggest a bug in the plugin, so if you want to continue using it then check for an update or report the problem to the plugin developer.
Ahhhh. I was thinking VST instruments. I completely forgot about the inserts. I don’t need the Youlean thing. It was just a handy reference. One of those free VSTs. I’m going to blow that off my system.
Thanks for the help. This is a rather unexpected cause/effect relationship. I bet there is something similar in most of those other complaints about really slow performance.
As far as I am concerned, the only real useful stuff that slows me down is the condense feature. We used to have different features introduced in the first versions of Dorico that were really slowing things down, and every time the team was able to fix that with optimization in the next updates, so I don’t expect anything else
I am done with installing free (or not free) stuff that I do not use, and I guess you’re going to follow this path as well!
Indeed. And I did report it to the Youlean people as Paul suggested. Youlean is a nice tool, but certainly not essential to the notation process. I had purchased the Insight VST from Izotope but I think YouLean is a bit easier to understand, at least for me.
It is temping to think of Dorico as a “notation variation” of a DAW. And I think it is completely proper to be thinking that way in terms of long-term convergence of technologies. But it is a rapidly changing field and we do have to be careful about unintended consequences along the way – and inviting problems we don’t need.
For the benefit of anybody in the future who stumbles upon this thread while trying to figure out why their scores are having intense performance problems, let me summarize:
Early in the project I had added Youlean to an insert in the mixer. It did what was expected, displaying the “mastered” levels in industry-standard (i.e. LUFS) terms. I saw no ill effects and gave it no more thought.
I proceeded to go many different directions with this score, adding and deleting flows, adding some players and all the other stuff one does during the creative process.
Probably 3 weeks after I had added YouLean, I started seeing really severe performance problems with this score. I had no reason to associate this with any VST instrument or effect. The symptoms seemed completely disconnected from anything the VSTs were doing, with one exception. When initially opening the Dorico file, Dorico displayed the progress bar about as fast as normal and displayed the notes within a few seconds – a normal time. But then the system would be locked tightly for about 3 minutes. At the end of the three minutes, the HSSE dialog would appear and then the score could be edited. So that much looked to be associated with VSTs or VSTis. I should note that the symptom was a very tight CPU loop that consumed one hyperthread (leaving 7 other hyperthreads on my machine mostly idle,) yet the entire PC was essentially locked up. In other words, this was a tight single-threaded loop that was dominating the memory system.
What was particularly confounding was that this same 3 minute lock-up continued to happen regularly during editing, not just at open time. I have no idea why this would happen, even knowing that the root cause was a massive build-up of VST data by Youlean.
It wasn’t until much later that I was able to eliminate practically all of the Dorico elements with the 3-minute problem remaining. That directed attention the the massive size of the Dorico file (26 MB for a score that had no flows and no players.) 7-Zip allowed me to look inside the Dorico container to see that the big storage was vstdata. And Paul was able to tie that to Youlean.
So if you are having very slow performance on a score, particularly with a tight single-threaded loop, there is a good chance it is related to a third party VST effect or VST instrument. In my case, one VST was saving a vast amount of data, but it is probably possible that a VST could get into a similar tight loop without storing an extraordinary amount of data in the Dorico file.
Lesson learned: Avoid all VSTs and VSTis that aren’t absolutely essential to your notation process, especially during the composition part of your project. If you need to add VSTs and VSTis in the later stages in order to accomplish your playback objectives, then do so carefully and make sure you retain good backup copies of your project before adding all those VSTs and VSTis. But don’t include those VSTs in the templates you will use when starting future projects.
Here is a response from the developers of YouLean:
“That is normal behavior since the plugin needs to save all graphs, so if measurement gets longer it will cause big projects. You can disable graph saving in the settings.”
I infer from that message that if one inserts Youlean in the Dorico mixer (and in any DAW for that matter), by default the vstdata will just keep growing.
====
On edit, I received a further message that the develop is looking into options to place a cap on the vstdata that Youlean will store. It is actually a nice little program, although not really essential to the notation process.
We’ve also identified a performance bottleneck that was much slower than expected as a result of looking into this. In our development build I can now open that score in 15s. The performance improvements should make loading and switching scores appreciably faster.
