I don’t have a way around this, sorry about that. But what’s happening is the coloring is actually based on the color’s position in the list of colors. When you add or remove Colors from the middle of the list it changes all the colors after it in the list. The only way I know around this is to only add or remove colors from the end of the list. Which in practice means you can’t readily remove them.
For example if you have blue as the 10th color in the list and red the 11th. Then if you insert a new color in the list at spot 5, everything after it will slide down 1 spot so blue becomes 11 and red becomes 12. And everything you had previously colored red will become blue.