Yeah, I think you may need to do some more system tweaking to be able to get your buffer down without audio dropouts.
See if there is anything missed here:
http://www.sweetwater.com/sweetcare/articles/pc-optimization-guide-for-windows-7/
With my UR824’s @ 512 samples I get 16.825 in and 19.819 out. But with my system and it’s setup I can record vocals on a fully mixed 60+track projects with tons of FX @ 128 (7.12 in and 8.118 out) without issues. 96 Samples (5.397 in and 6.395 out) isn’t a problem for tracking audio without many FX going on.
See what those setup steps get ya and report back. If that doesn’t work maybe there is some conflict with graphics driver or something.