Channel link button alters sound

I am trying to record some old vinyl with a UR44C using channel 3 for left and channel 4 for right.

If I use the channels separately and pan hard left and right respectively the sound is completely different to when the channels are together with the link button.

Am I misunderstanding something simple?