When rehearsal marks were first implemented in Dorico, they used a font style rather than a paragraph style. Font styles are simpler, and don’t allow rich text formatting, nor do they allow different sizes in score and part layouts, and nor do they directly support enclosures of any kind.
Enclosures for rehearsal marks also need to work a bit differently than enclosures for regular bordered text because you typically want to enforce a minimum size for the enclosure, regardless of the content of the mark. Text doesn’t do this, but the special routines we’ve written for rehearsal marks do. Similarly, we don’t support circular enclosures for text, but we do for rehearsal marks.
If you want to use the options defined in the paragraph style instead of the ones defined in Engraving Options, you can do so: set Enclosure type to No enclosure on the Rehearsal Marks page of Engraving Options, and then make the desired changes to the rehearsal marks paragraph style.
But I would recommend using the enclosures as specified in Engraving Options, for the reasons outlined above.