There’s a bug with this that’s been discussed in the last couple of days, here.
Are you actually deactivating it, or are you switching to Play mode and seeing that it appears to be turned off? From it’s “off” state, try clicking it twice. Now it should actually be off.