OK, thanks. Will report back asap. Charles Manning escribió: > On Thursday 04 March 2010 04:17:53 Hugo Etchegoyen wrote: > >> Charles, >> >> Thank you very much for answering. In the meantime I discovered there is >> a yaffs list and I've subscribed to it, so if you prefer to continue >> this thread in the list just advise. >> >> It looks that you are already addressing my issues, so you can count me >> as a guinea-pig. My timescale for this project is 2-3 months. >> >> To be a little bit more precise about my application, I will try to give >> you some timing figures. They are not very precise because my test board >> is not working right know so I have to count on my memory, but I think >> you'll get the idea. >> >> I have some 200 millisecs available for shutdown. I can save state and >> sync yaffs in less than 100 millisecs if GC doesn't interfere, but when >> it does I can easily run out of time. I did some profiling and realised >> that the execution time of yaffs_GarbageCollectBlock() could climb to >> some 300 millisecs, depending on how many chunks were copied at a time, >> so I came up with the idea of a smaller value for maxCopies. I guess >> your proposed code is safer than just setting maxCopies to 3. >> > > 300ms is excessive for gc. I will have a look to see if there is something > interfering. > > >> Background GC would be great since it would improve the average writing >> speed, as most writes would not involve in-line GC. However, since the >> probability of entering GC at power fail would be much smaller, but not >> zero, I think it would still be necessary to decrease maxCopies in order >> to cover the worst case. >> > > Yes indeed, background gc would not block for long. > > >> Your idea of updating directory times in >> background is also very nice. My application doesn't care about file or >> directory times, so I will mount yaffs with "noatime" and "nodiratime" >> in order to increase performance and flash endurance. >> > yaffs does not support atimes because it is expensive to do a header write for > eacyaffs is already noatime and nodiratime whether you want it or not :-). > > The directory times I'm talking about are the directory mtime and ctime that > get changed every time the directory is changed so that client apps can know > to refresh their view etc. Every time a file is created, renamed or unlinked > we write a directory entry reflect the change. This doubles the number of > writes assocaited with creating or deleting empty files. > > The new approach will put the "dirty" directories into a dirty directory list > and use the background thread to periodically (every second or so) update the > directory times. That should reduce the number of directory update writes > significantly. > > This is particularly bad when running test code which, say, creates 1000 files > in a directory then deletes them resulting in 2000 directory update writes > (ie. 4000 writes in total), half of which only last for a very brief time. > With the new code there would only be a handful of directory update writes. > > > >> I agree that refreshing old data looks much better than refreshing >> random blocks and at the same time does static wear leveling, so as soon >> as you have code for this (or any hints) I will gladly try it. >> > > It is in cvs. Give it a go. > > >> Cheers, >> Hugo >> >> Charles Manning escribió: >> >>> Hugo >>> >>> This is interesting as I am looking at implementing background gc and >>> revising the gc policies. What are yourr timescales? I'd be interested in >>> someone being a bleeding edge guinea-pig. >>> >>> On Tuesday 02 March 2010 09:41:45 Hugo Etchegoyen wrote: >>> >>>> Dear Charles, >>>> >>>> I intend to use yaffs2 in a uCLinux embedded application, but I need >>>> some tuning. >>>> >>>> My hardware detects power failure and gives me a short time for >>>> shutdown. >>>> >>> How short is short? >>> >>> >>>> During this time I must save state to flash. This involves a >>>> few small writes totalling much less that the size of an erase unit. >>>> Then I sync (and unmount?) the yaffs partition. >>>> >>>> As there is little time available for shutdown, I need to make some >>>> modifications to garbage collection: >>>> >>>> 1. I must be able to disable GC during power fail so that it will not >>>> waste valuable time. >>>> 2. GC must be more spreadout than it is now, i.e. >>>> yaffs_GarbageCollectBlock() should only copy a few blocks at a time, >>>> whether aggressive or not, so that power fail will not have to wait too >>>> much before disabling GC. >>>> >>>> As an additional requirement, my flash will be pretty full. It will have >>>> lots of static data and the dynamic data will be very hot, which is a >>>> bad case for endurance. For that reason I also need to: >>>> >>>> 3. Add static GC to yaffs. >>>> >>>> I think (1) can be easily satisfied by raising a flag on power fail and >>>> having yaffs_CheckGarbageCollection() check the flag and do nothing if >>>> it is set. >>>> >>> At present it is quite easy to defeat gc by just setting dev->isDoingGC >>> to 1. >>> >>> >>>> Regarding (2), I have already tried a simple modification changing line >>>> 3036 of yaffs_guts.c, in function yaffs_GarbageCollectBlock() as >>>> follows: >>>> >>>> Original: maxCopies = (wholeBlock) ? dev->nChunksPerBlock : 10; >>>> New version: maxCopies = 3; >>>> >>>> I've tried this under rather heavy load and everything seems to work >>>> properly. I hope a figure of 3 guarantees that we will not run out of >>>> free blocks, because during GC three chunks are freed for each new chunk >>>> written. >>>> >>> There are some conditions where this policy might fail, but I would >>> expect 3 to probably work. >>> >>> Perhaps >>> maxCopies = (wholeBlock) ? 5 : 3 or something like that would be safer. >>> >>> >>>> Regarding (3), I am thinking of modifying >>>> yaffs_FindBlockForGarbageCollection() so that once in a while, let's say >>>> one time out of 100, it will just choose a random full block. This >>>> should tend to move static data around. >>>> >>> It is potentially dangerous to just randomly gc a block unless you also >>> check it is not disqualified. >>> >>> The new block refreshing code will also do what you're wanting. It >>> infrequently gc's the oldest block thus rewriting static data. >>> >>> >>>> I'm aware that all the intended modifications tend to postpone or slow >>>> down GC. Do you think it would be advisable to increase the value of >>>> nReservedBlocks in the yaffs device to compensate, and by how much? >>>> >>> nReserved blocks of 5 is already pretty conservative. The dangers are: >>> a) The possibility of a block going bad during gc means we need to keep >>> some extra handy. >>> b) With the current gc logic, nErasedBlocks should never drop below 4, >>> maybe 3. maxcopies = 3 might make things a bit worse because a gc can >>> sometimes end up with very little reclaim (no reclaim is a possibility). >>> That might mean we lose one block of headroom. >>> >>> Thus from a quick mental audit, it sounds like the current level is safe. >>> >>> I might be inclined to do something like: >>> if(dev->nErasedBlocks < 3) >>> maxCopies = dev->nChunksPerBlock; /* Should never happen. This is just a >>> safety net if erased space get really low.*/ >>> else if(wholeBlock) >>> maxCopies = 5; >>> else >>> maxCopies = 3; >>> >>> There is currently an ugliness caused by "shrink headers" which constrain >>> the gc order meaning that often we can't collect the blocks that would >>> give the most free space. Sometimes these end up in "chains" meaning >>> there is a sequence of these gc's which causes gc to grind hard. >>> >>> As I mentioned at the top, I am revising gc in two ways: >>> 1) Making it less "lumpy" by doing more smaller gcs. >>> 2) Adding a background thread to do gc behind the scenes meaning that it >>> should only very seldom need to be done in the writing thread. The >>> background thread will also do directory time updating which should cut >>> down on the number of writes when creating files etc. >>> >>> The background gc will target: >>> 1) Trying to keep at least 25% of space in erased blocks. That means >>> writes should happen fast. >>> 2) In-write gc will only happen if erased space gets v low. >>> 3) Eliminate or reduce those pesky shrink header chains so that if gc >>> does happen in the write thread it doesn't become too onerous. >>> >>> -- Charles >>> > > > > > -- Ing. Hugo Eduardo Etchegoyen* *Gerente Dto. Software de Base Compañía Hasar | Grupo Hasar* *Marcos Sastre y José Ingenieros El Talar. Pacheco [B1618CSD] Buenos Aires. Argentina Tel [54 11] 4117 8900 | Fax [54 11] 4117 8998 E-mail: hetchegoyen@hasar.com Visítenos en: www.hasar.com Información legal y política de confidencialidad: www.grupohasar.com/disclaimer