I see they can be added, but what about taking them out of a custom score order?
You don’t add or remove families directly in instrument score orders: the families that appear in the dialog appear based on the families in which the instruments in the score order appear. In the dialog, the families list really exists only to make the overall list easier to work with, and allows you to move entire blocks of instruments above or below other blocks.