I think any tempo adjustment will leave the pitch wrong. I think you need to do tape style pitch adjustments until the tempo is constant. Maybe even put the audio in a sampler track and use pitch bend to fix it. Generating a tempo map can help show you where you need to make your adjustments.