When a system is overfull, it’s much more likely to create what we call a “minimum distance violation”, which means that Dorico needs to move two consecutive spacing columns further apart than their ideal distance because something would otherwise cause the two columns to collide. In the normal run of things, you’d typically see this only with lyrics, but when you have tightly spaced music with lots of accidentals or other left and right protrusions, you can run into these kinds of violations in other situations. Whatever the cause of the violations, this means that Dorico has to move the columns further apart than the ideal distance for a note of that duration.
Dorico can do this until the system is 100% full, because any extra space that is left over can be allocated to some of the columns to make the system exactly 100% full. As soon as the system is more than 100% full, it has to take a different approach: it still tries to maintain the minimum distance violations so that things don’t collide, but it then has to take away space from all of the non-violating columns, with the result that the overall result is very different.
The only alternative to this would be for us to add a mode to note spacing that always disregards all minimum distance violations, i.e. that will always space the music according to the ideal values defined for note spacing, which would mean that Dorico would not create extra space not only for lyrics but also for e.g. accidentals on the following column. The effect of this option would be that Dorico would space the music precisely according to the rhythms, not making extra space for collisions between things like backnotes, accidentals, lyrics, etc. But the advantage would be that you would find the transition between < 100% and > 100% full completely smooth, i.e. nothing would move unexpectedly as you cross the threshold.
There is no “third way” where you can still have all of the advantages of Dorico’s automatic collision resolution and then magically not have a transition between two approaches that can create changes that you cannot anticipate.
We have already implemented the option to space the system ignoring minimum distance violations in our development builds, so this option will be available in future versions of the software. I must stress that on the whole I would not recommend using this new option unless you have very specific requirements.