If you buy Cubase 8 Pro, that automatically gives you the licenses for all previous versions like VST5 & SX3, but the licenses would be on the Dongle.
I don’t know if the old cubase programs will run on your iMac, other Mac users will have to chime in.
I would confirm that they will run first before buying C8, as, unless you can add USB ports externally on your old machine, I’m not sure how you would even get VST5 to run on the old machine, as if you bought C8, the licenses would be on the Dongle.
Sounds confusing I know, but I’m not a Mac user so I’m just going by what I know from a PC perspective.