Dumb question maybe, but do you have a key signature specified in the piece? I catch myself forgetting to put a key signature when working in C or a minor. I don’t think you can specify the key signature option in the transpose dialog if you don’t have a key specified.
Sorry if I step in here, this usually happens, when you have no key signature specified. Try to add a key signature by Shift-KCEnter and try to do the transposition again.
@konradh, I just thought to retry replicating the error, and I may have it! Instead of SHIFT-KCENTER I used the Independent Key Sig entry SHIFT-KCOPT/ALT-ENTER. In that case the *Transpose Key Sigs option is grayed out, which I think makes good sense.