I don’t profess to know much of anything about ASIO or how it works per se, but the driver for your audio interface is also software, so it should benefit from extra processing power. E.g. for buffer processing. Even though the VST Performance meter doesn’t equal CPU usage, it should be affected by differentials in CPU resources available.
Another way to potentially gain ground against resource loss, is using better plugins. Some plugins are just written better than others, and results in less resource usage. Some are irresistible I know, but when it comes down to crunch, sometimes it’s just beneficial to shop around.
There are a number of plugins on the market that are written using VST frameworks (i.e. wrappers of various purpose), some which are modular prepacks and others that are language or API bridges of some form or another, and such architectures come with more or less management cost. If you then have a lot of instances of these plugins, the overhead could be crucial.
An simple example of a plugin affecting the VST Performance meter can be seen in Steinberg’s own HALion 5. I pick it H5 because I use it a lot, it’s Steinberg’s (for less upset minds) and it can be used for a demonstration. If you pick a synth sound (no samples used) that does not utilize the unison functionality, play a piece and look at the VST Performance meter, and then add unison with 8 voices per note, and do the same reading again, there is a difference.
Does this mean H5 is poorly written? Of course not, it’s likely written better than many others, but it does mean that the nature of a plugin can affect the VST Performance meter. So compare the plugins you use and the way you use them, because a simple parameter within a plugin can make a difference, and if you use many instances of a particularly setting, you could be chasing that meter for the wrong reason.