I’ve got two basic answers to the question on when a mix is finished, probably with a whole bunch of sub-answers. 
One answer is in the case of a deadline, such as for any specific pitch opportunity where, if it’s not submitted by such and such a date and time, the song/recording doesn’t get considered. For that case, it is whatever the best I can do to get the submission in ahead of the deadline is what is “done” (for now, at least). If I can’t get something that is at least “good enough”, then I won’t be able to submit, but sometimes “good enough” may not be ideally what I’d want to pitch if I had more time.
This happened to me very recently with an opportunity that needed an original Christmas song for a movie opportunity, via a music library. Writing the song took me longer than optimal, and that put a lot of time pressure on getting something recorded in time. What I submitted was somewhere between what I’d have normally considered a work mix and a rough mix, but it still “felt like a record”, and I got it in maybe an hour or two before the deadline. The good news is the library signed the song/recording – whether they ended up pitching it to the movie company or not I have no clue. BUT I knew at the time that I wouldn’t want to release that recording as a single as it stood. There were a number of issues I recognized to be compromises that basically just kept “mixing as you go” aspects that were meant to be placeholders until I had time to do “a real mix”, including the lead vocal processing, lack of mix automation, and more.
The other answer is in the case of something I feel good enough to release, be it as a single or part of an album, where there generally aren’t fixed time limits, though there may occasionally be some (e.g. if it is a holiday release that either has to be in distribution by some latest date or wait until a year later). Over the course of working on a recording, I tend to do a lot of mixes. Some are just work mixes, where I don’t have all the elements of the arrangement in place, and I’m using temporary processing (especially on vocals) just to more or less make it feel like a record in order to evaluate the production in process as I add components. I tend to do car listening tests whenever I get to a breaking point (and have a conveniently-timed need to drive somewhere, even if just my weekly grocery shopping trip). Others are rough mixes, where I’m starting to work toward a final mix, but with recognized components I know I’ll need to work on later – this is often especially the case with lead vocal processing, where I may start out with a single plugin scenario then break things out to multiple buses, including possibly automating some elements. One key difference between what I consider a rough mix and a final mix candidate is fader (and sometimes effects) automation, which I tend to do relatively late in the game after “getting the sound” on all tracks and submixes at a basic level.
In this case, I keep doing listening tests as I’m going, making notes from those tests, until such time as I either don’t have any more notes or decide that whatever I do have notes on will have to do because I can’t do any better than where things stand at that point, and a project can’t go on forever. It’s got to go out the door at some point.
In the case of the Christmas song I alluded to above, the mix the library signed was the third mix. That was actually a far lower number than might be typical, in large part because there was a very short deadline, and I was running up against time just getting the arrangement together. The one I ultimately ended up dubbing as final for an upcoming single (releasing December 1st) was the sixth mix. That was also a relatively lower number of mixes than my typical case, in large part because the mix that I pitched to the library was already sounding pretty good, just not something I wanted to release. But I’ve had some more challenging projects go upwards of 20 mixes to get to final – I think my record may be 27 for one of the songs on my most recent album (Chasing Rainbows).
What’s the difference between some of those late-stage mixes? Typically, it is that there is something that is bugging me. One thing that turns up more often than other things in car listening tests is bass resonances and/or conflicts between the kick drum and the bass. I use a number of tools in my (small bedroom home) studio to try to catch what I can in different monitoring scenarios, but one thing I somehow seem to miss more often than other things is bass resonances that vibrate the speakers in my car. Other times it might be something like the balance of the vocal versus the instrumental tracks, or something like an instrumental solo that is either not quite loud enough or a little too loud. Or maybe some weird resonance on the lead vocal. Or…
It’s not that the recordings sound bad at that stage. It’s just that something is bugging me, and I make notes then try to figure out how to address it, preferably without damaging something else in the process.
FWIW, I’m using all virtual instrument tracks, with the exception of vocals, but I am trying to have my recordings sound like an actual band most of the time. So, for example, guitars are virtual guitars that I’ve played on a keyboard and/or programmed.
I tend to use the Waves NX plugins (e.g. CLA NX, Ocean Way Nashville NX, etc.) for “listening in various environments” to help me get a sense of how things will sound on different types of speakers (the CLA one emulates a boombox in addition to other speaker types), in addition to listening on my monitors, my computer speakers, and just straight headphones (without the environment emulation). Those tools have helped me cut down somewhat on the typical number of car listening tests I need to get to a final mix. I’ll also listen at different volume levels (e.g. taking it down about -18 dB from my normal level) to try and ferret out balance issues, and also go to another room to see how balances work when not as near to my speakers. There are always tradeoffs, so it’s a matter of picking and choosing. I also use visual tools, like iZotope Tonal Balance 2, to help me “see” issues my hearing may not be catching. And I do listen to reference recordings alongside the recording I’m working on at the moment (using the same monitoring tools).
Perhaps, at some level, there is also some influence from just wanting to finish a project so I can move on to another.
