Issue with WAV files on SD cards in Windows 10

If you use SD or CF cards and have (against the advice) upgraded to Windows 10 you should read this:
Windows 10 corrupting audio files
Now, it’s not quite as bad as that headline might suggest as it only appears to be affecting some users if all of the following are true:
– you have WAV files on an SD or CF card (e.g. as used in portable recorders)
– the card is formatted as FAT32
– you have upgraded to Windows 10 (and not clean-installed)
– the card reader is connected via USB (and some “internal” readers are connected internally via USB)

These are the links to the Microsoft Community forums which describe the issue:
Windows 10 corrupts files without warning on SD & CF memory cards (FAT32)
… and the longer, but harder to read one:
transferring .wav audio files

The scary bit is that, according to users who have experienced this issue, all that is required is for the card to be put into the card reader for the corruption to occur, i.e. you don’t even have to have tried to read from the card.

So for the moment the advice (apart from, “don’t upgrade to Windows 10 yet”) to write-protect your cards before putting them in a Windows 10 machine!

The key takaways from that are:

a) The file header has been overwritten, but the audio content is OK. The first four bytes need to be replaced by ‘RIFF’, but you need a hex file editor to do that. Notepad++ might be able to do it.

b) MS is working on a fix.

Well. No problem here.

My steps:

  1. Inserted a 2GB SanDisk SD card into my W8.1 desktop computer, using a SanDisk USB card reader.
  2. Formatted the SD card as FAT32 with default 4k sector size.
  3. Copied a 391MB Cubase 192k 32bit float wave file to it. It played OK in WMP.
  4. Inserted the SD card & reader into a Win10 tablet.
  5. Double-clicked on the file. It played OK in Groove.
  6. Re-inserted the SD card & reader back into the desktop computer. It played OK in WMP.

Now, just to test how the header could be edited, on the SD card, I:

  1. Opened the wav file in Notepad++.
  2. Replaced the ‘RIFF’ at the start with ‘XXXX’, and saved it.
  3. WMP indicated the file was corrupt.
  4. Re-opened the file in Notepad++.
  5. Replaced the ‘XXXX’ with ‘RIFF’, and saved it.
  6. WMP played the file without problems.

I haven’t been able to confirm either. Looking at the latest updates on this, it’s beginning to look like it only occurs with WAV files that have been written to SD or CF cards by Sound Devices equipment! Nevertheless, at the time of writing Sound Devices still have the “technical alert” on their website.

I’m really curious now because this is truly bizarre, but for the rest of us, I think it means “relax”!

This has been diagnosed now and it is confirmed that Sound Devices will issue a firmware update for their recorders as well as Microsoft issuing a patch for Windows 10. You can read the detail of this strange and unlucky combination here: Technical update on the Sound Devices / Windows 10 memory card issue

Actually I’m impressed with MS here.

They could have just said that Sound Devices incorrectly set an MS-reserved bit in the file header, and left it at that, leaving Sound Devices to clean up their own mess that they created for their customers.

But they are going that extra to make sure their customers are protected from errant third-party designs in future, at least in regard to use of the encryption bit, by checking that the file is actually encrypted.

I don’t know how far any software maker would want to double-check correct use of every protocol header bit against the content itself, as it could add substantial overhead to handling, and might open up new attack vectors, just because of assumptions that may be made in that handling. After all, it is straightforward to assume that invalid use of header bits makes reliance on the rest of the file being valid unwise.

For example, in my testing of editing the first four bytes of a wav file, changing them from ‘RIFF’ invalidated the file. Should EVERY piece of software double-check the content to see if the file was a valid wav file anyway. That would undermine the purpose of the headers in the first place, which is to define how the content is to be interpreted.

This is the original transferring .wav audio files thread on the MS forums where the problem was raised.

Psychlist1972, the commenter on that GearSlutz thread cited above, is an MS staffer, and his comments were cloned from the MS thread.

Pete (Psychlist1972 at GearSlutz) at MS has given an excellent reply to my query.

Basically, the problem was that the Sound Devices’ firmware erroneously set the MS-reserved directory file entry bit now being used for indicating encryption in FAT32 systems. They have an internal reason for it, but they should have read the FAT32 specs before relying on it.

Now, as Pete indicated, MS will be checking that the header bit in the files themselves do in fact match their corresponding directory entry bit, which is low overhead, as he confirmed, because it does not involve reading the file’s content data itseslf.

He indicated that MS was going to do this because they didn’t know what time frame Sound Devices could work to, so they didn’t want MS clients to be put out any more than needed, if they could do any thing about it. MS can’t rely upon people updating their firmware either.

The interesting thing about that is that he thinks MS will move faster than the OEM, which is a good indication that MS is not in a ‘feet dragging’ mode any more, but are being a lot more proactive and pragmatic, which gives us hope for the W10 issues with Cubase.

Pete at MS has indicated that an MS patch will be available very soon.

He has produced PowerShell scripts that can be used in the interim. See the MS thread liked above.

Pete indicates that the fix from MS’s end will be available in about a week.

Note that while the original problem was solely due to Sound Devices (SD) improperly using FAT32 directory file entries, MS took it upon themselves to install the double-check (which is useful in its own right, beyond SD) so that SD users who hadn’t installed updated SD drivers (whenever they appear) would not be disadvantaged.

Just noticed what the Sound Devices site writes about this issue, and it is a total cop out on their behalf.

In regard to the reserved bit that they incorrectly used, they write ‘Prior to Windows 10, Microsoft did not use this bit, which was used by Sound Devices, so Quality Assurance testing would not have revealed any conflict.’

This is total rubbish, because it was the design team’s obligation to know that was a limiting requirement, and their software quality assurance team to double-check that the requirements were correct and implemented correctly.

Futhermore, they imply that MS provided a fix for their specific issue. In fact, MS provided a patch that will help prevent other slack manufacturers from using the bit detrimentally. In other words, the issue was their own incompetence, and MS is trying to protect the clients of other makers from similar incompetence.

Specifically, the patch makes sure that if the bit is set, then a check is done to ensure that the contents of the wav file are likely to be actually encrypted (by checking the first four bytes for the signature for such encoding), before treating the file as encrypted. Obviously, it cannot really be done except by examining the file until explicitly invalid data occurs, which is impractical time-wise, given the nature of the type and amount of data involved. It is a simple double-check that should not be necessary, except for some makers’ lack of basic diligence, but cannot protect against makers who are doubly slack (use the reserved bit incorrectly AND use a wav file header that signifies encryption content incorrectly).

It is a shame that such companies make a mistake, and then falsely blame it on others, and then take protective and preventative action by those others to be remedial action and an admission of fault, thus absolving them of culpability.

By this, I would not trust Sound Devices at all, because when the crunch comes, they will attempt to negate their obligations.