Hello Charles, et.al.,
I could use some advice on getting Background Garbage collection to behave better, to avoid stalling YAFFS2 operations.
Have a YAFFS2 to an EMMC NAND-flash module with about 7GB of room.
Block size is about 8MB = 256 x 32KBchunks.
Have 930 blocks.
We call the background garbage collector every 5 seconds.
We have a guaranteed reserve space of over 256MB. (Validated by examining the pages_in_use elements).
We generally write files into this system until it gets close to full and then delete the oldest file. Typical file sizes are 40MB. In past attempts to keep YAFFS happy we've ensured that we always have 256MB of free space and delete an old file to keep that amount. That's on top of any free blocks maintained inside YAFFS. Despite theoretically always having lots of room for new data we experience yaffs-stalls often. We write a nice steady 11KB/sec or so, in 16384 byte writes using yaffs_write(), along with other random reads, writes, directory listings, etc. Typically these operations take very little time. But "randomly” we find that a single operation will suddenly "stall" for up to 6 SECONDS. This lack of predictability has caused numerous problems.
What I've found is that the stalls are accounted for by the aggressive garbage collection reclaiming an 8MB block that is almost entirely in use. YAFFS copies all of the active chunks to a new block, in a single operation - hence the 6 seconds. Furthermore, there are other gc-candidate blocks that have far fewer chunks in use. Reclaiming those blocks would involve far fewer chunk-copies, but the garbage collection scheme has a mind of its own about what is appropriate to use. Furthermore, the background garbage collection, which could be reclaiming this data, usually stops doing anything for about a minute between garbage collection cycles.
Part of this is embedded in function yaffs_find_gc_block() which picks out which block to collect next. I have a number of questions about that function that I can't easily discern from reading the code.
Don't know what the Prioritised scheme does. Doesn't seem to be activating in my system. I could use a definition of "Prioritised".
We try to scan a range of blocks, looking for a block we can reclaim. If not in aggressive mode, I get a threshold of 4 chunks, which I believe means we'll only accept FULL blocks that are still using 4 or fewer chunks. That's a pretty restrictive threshold. Basically we'll only reclaim blocks that are mostly deleted, except when aggressively (all at once) reclaiming. Was the idea that we hope that data will be deleted so that Passive mode can collect the block, before we hit the Agressive wall, with the accompanying stall?
We won't use a block unless it passes yaffs_block_ok_for_gc(). That function appears to test that the sequence# of the selected block must match or be older than the oldest_dirty_seq number. [I could be wrong about this since a block with has_shrink_hdr would be allowed, but I don’t fully understand that. We don't generally shrink files so I believe that shouldn't affect my situation.] If I understand the oldest_dirty_seq mechanism, there is only one block with the oldest_dirty_seq. All other blocks will be "newer". Thus, the only block that could possibly be used for GC will be the oldest_dirty_block.
Am I reading that correctly?
This leads me to wonder why we go through the loop in the first place. Seems like it's all a waste of time, since there is only one block that will pass the sequence test. And in Passive mode, we only check 100 blocks per call. With 930 blocks, we can only hit the oldest reclaimable block every tenth passive GC call. My system calls background GC every five seconds, so that can be a delay of almost a minute to find that block.
Eventually these restrictions hold off the system enough that the final clause kicks in to "try the oldest dirty block because that's gumming up the works". And that's only after 10 attempts (again, in my case 50 seconds). This seems to be the only method my system uses to select a gc-block!
This all makes Background garbage collection problematic. We have some time to gradually transfer chunks from an old block, without burdening yaffs operations with stalls, but that has to be passive to avoid a stall. In Passive, we won't collect anything unless it is almost completely unused. And we won't accept anything but the oldest sequence dirty block, which might not meet that criterion. And we might not even find that block for many cycles due to searching only 100 blocks at a time.
The result of this is that my system only reclaims blocks in oldest_dirty_seq order, regardless of how used that block is. The process holds off so long that the background garbage collection is unable to finish the gradual copies before we run low on fully erased blocks. We switch to Agressive mode at some random time and stall while the copies complete, hanging the file system for several seconds.
What I have is a system where the hardware is capable of writing 8MB in five seconds (as demonstrated during the stall periods), which is only being asked to log 10KB/sec, but which hangs often for long periods causing backups elsewhere in the system, poor user interface response times, etc. All this despite having thrown a lot of free space at the problem.
- Have I misread this code?
- Is there some "tuning" I could do to reduce/remove the stalls?
- Am I going to have to rewrite this code to better handle my specific situation?
- Have I mis-interpretted the requirement to only reclaim blocks based on sequence number? If not, then what is the purpose of that requirement? If it is associated with wear-leveling, then my EMMC hardware handles that and I don't have to keep that restriction.
Based on the pages_in_use counts, there are a number of blocks that could be garbage collected to regain significant erased chunks. But the garbage collection usually appears to be collecting blocks without much positive impact. In fact, the collection of these mostly used blocks quickly fills the current allocation block, leading to aggressive GC and a guaranteed stall. Unless the passive GC is able to keep up, it doesn’t do much good.
Any advice would be appreciated.
Thanks,
Tom Coates :::/
Trimble Navigation
(text/plain)