Thanks, Charles!

So, if we run out of available space, I'll get an error when I try to write to the hole? That should be OK... Very neat.

I think I know what my real problem is. I write the expected file size to the file header. This enables me to recover from a failed queue expansion: https://github.com/square/retrofit/blob/master/modules/android/src/retrofit/io/QueueFile.java

In the case I'm investigating, the header says 16k, but the file size is only 4k.

I used "rwd" mode which syncs data by not metadata writes. In other words, it uses fdatasync() instead of fsync(). I think the file size qualifies as metadata. It sounds like my header write synced to storage but the file size metadata didn't, even though it was written first.

If I turn on metadata syncing, ftruncate() should sync, and everything should be OK.

Sound plausible? 

Thanks,
Bob