I tried to transform a part from G#maj to Ab maj, this is what happened:
…How do I avoid this (see screenshot)…enharmonic changement doesn’t work… the sharps don’t become flats.
no image visible
Transpose the appropriate notes up a diminished 2nd.
There are a variety of Respell options too, either manual or automatic. I would imagine if you selected the region and chose Respell Notes Automatically it would work, but you can always do it with a manual shortcut too. (I think I changed my shortcuts, so yours might be different.)
Since I see an A♭ major key signature, I suppose you may have selected just the key signature, expecting Dorico to transpose all the following music like Finale does. As you’ve seen, Dorico doesn’t work that way: it operates only on what is selected; Sibelius likewise. This has the advantage that you can change the key signature without disturbing the notes. The subsequent music is rewritten to include any necessary accidentals.