Custom key signatures can be transposed, provided the interval you want to transpose by allows a valid set of note spellings and accidentals. Dorico won’t allow you to specify a translation that will result in invalid spellings or accidentals, so if you find that you cannot choose an interval that you expect in Write > Transpose, that will be the reason: the interval you want to transpose by is invalid for the tonality system you have created.