Okay, I see what you are talking about now on the button:
Darn obscure-looking buttons.
Anyway, I just tried it now (in 12.0.10 Pro). I actually didn’t have the track armed for recording. When I first clicked on the button (I was only selected on a lane of the track, rather than the track header itself), it warned me to arm a track for recording or select a track. (Perhaps I was actually selected on a marker track or something other than an active instrument track at that point and selecting a lane of the track in question didn’t change the selection?) But once I selected the track header for the track in question, the notes I’d played were put into the track, albeit overlaying the top lane, rather than in a new lane. (I’d noticed a similar thing yesterday when using retrospective recording in that the notes went over an existing lane, though they separated out once I did a clean up lanes command – the new recording ended up in the first lane of the track rather than the previously empty lane.)
After I did that, I looked at the RR pulldown in the track inspector, and it was grayed out, but the button on the transport was not. So I did another similar exercise, playing over a part, and this time I could put the RR info in the track from the inspector, but the inspector version gets grayed out right after doing that, whereas the transport button does no (and still seems to work for me, I think)
Even more interestingly, if I used the track inspector function to empty the RR buffer (while it is still enabled), it gets grayed out, but the transport RR button does not and still works for inserting MIDI data (I did not check to see if it was what I played – I was really just playing garbage then undoing after the inserting to test this).
I guess the bottom line in my case (and I’m on Windows 10, FWIW), is the retrospective recording seems to work okay for inserting data, but the behavior of the button versus the pulldown (which is all I’ve ever used previously) versus the empty buffer command is at least inconsistent. One key, though, is that the track you’ve got selected is not frozen or deactivated, and is an instrument track (at least in my case – I am not using multiple outputs from VST instruments nor any external synths), and not a just a lane in such a track (perhaps with another non-MIDI track being the active track). But, when I tried inserting the recording in that latter case, it gave me an explicit warning.