It doesn’t need to be fast. The hardware layer of MIDI is after all a pretty simple serial interface running at 31.5 Kb/s It just needs to be accurate. Which is why the old Steinberg Midex 8x8 had clever stuff™ built into the driver that timestamped the MIDI events and told the host what the relative timing of MIDI events SHOULD be.
The only MIDI interfaces that still has such a thing (as I understand it) is the MOTO range, where they have MTS (MIDI Time Stamp) built in.
The reason why nobody cares (much) about this these days is that the MIDI data flying between virtual instruments and hosts can actually be sample accurate, and nobody (except old prog-rockers like me with a room full of MIDI gear and a picture of Rick Wakeman on the wall) uses stuff with those 5 pin DIN leads any more.