This one trips me up frequently. I’m trying to select the handle, but I get the spin box instead. I have to click on the far right side of the handle to select it. It seems that the invisible “click boundary” for the number field extends to overlap the handle.
Once selected, I do use the arrow keys to move between handles, but obviously the mouse is still necessary for the first selection.