The part of the dialog with the octave is just a helper to calculate the interval of transposition. Octave 4 is the octave above middle C, so if you were transposing from Bb to C, and wanted to transpose up a major 2nd you would specify Bb octave 4 to C octave 5. As opposed to transposing down a minor 7th: Bb4 to C4. Pressing the Apply button there puts the appropriate values in the Quality and Interval boxes. If one already knows those values the calculator isn’t necessary.
I don’t use transpose often enough to trust myself on exactly what the other options do. The documentation would definitely help there.