Thanks, Charles!

So, if we run out of available space, I'll get an error when I try to write
to the hole? That should be OK... Very neat.

I think I know what my real problem is. I write the expected file size to
the file header. This enables me to recover from a failed queue expansion:
https://github.com/square/retrofit/blob/master/modules/android/src/retrofit/io/QueueFile.java

In the case I'm investigating, the header says 16k, but the file size is
only 4k.

I used "rwd" mode which syncs data by not metadata writes. In other words,
it uses fdatasync() instead of fsync(). I think the file size qualifies as
metadata. It sounds like my header write synced to storage but the file size
metadata didn't, even though it was written first.

If I turn on metadata syncing, ftruncate() should sync, and everything
should be OK.

Sound plausible?

Thanks,
Bob