[Yaffs] Why does YAFFS skip the first block?

[Peter Barada]

On Wed, 2005-06-22 at 16:55 +1200, Charles Manning wrote:
> 
> I have thought of a very simple way to add the mapping stuff so that it is 
> done completely internally to YAFFS, in a very non-obtrusive way (ie. I don't 
> want to do something that potentially hurts what is already there). YAFFS 
> would then be able to accept and use block zero, but would work completely 
> transparently if a non-zero starting block was used.
> 
> I think Frank's main point is that doing the work around (for YAFFS) requires 
> hacking the mtd access stuff. ie., It seems nuts to go hack somewhere else to 
> fdo a YAFFS workaround. For the most part, people  using YAFFS on NOR have 
> done this with RTOSs, and had to write their own flash access functions so it 
> was not a big deal.
> 
> While the loss of a single block is not significant for NAND systems it is 
> significant for many NOR systems. YAFFS is a NAND system, but there have 
> already been some hacks done to make YAFFS NOR friendlier.
> 
> I consider my arm twisted and will do/test this in the next couple of days.

I'd like to help, if I can.

> On Wednesday 22 June 2005 16:09, you wrote:
> > On Wed, 2005-06-22 at 13:06 +1200, Charles Manning wrote:
> > > On Wednesday 22 June 2005 12:13, Peter Barada wrote:
> > > <snip>
> > >
> > > > > All is not lost though, there is a way to use that block zero,  and
> > > > > it is pretty straight forward. You just need to tell YAFFS some lies!
> > > > >
> > > > > All yaffs flash calls go through a few access functions.
> > > > > All you need to do is to do a mapping in these functions so that
> > > > > yaffs block 1 = physical block 0.
> > > > >
> > > > > eg.
> > > > > int (*writeChunkToNAND)(struct yaffs_DeviceStruct *dev,int
> > > > > chunkInNAND, const __u8 *data, yaffs_Spare *spare)
> > > > > {
> > > > >       chunkInNAND -= dev->nChunksPerBlock;
> > > > >
> > > > >      .....   // rest of stuff
> > > > > }
> > > > >
> > > > > int (*eraseBlockInNAND)(struct yaffs_DeviceStruct *dev,int
> > > > > blockInNAND) {
> > > > >     blockInNAND--;
> > > > >     ...// rest of stuff
> > > > > }
> > > > >
> > > > > Now you must need to keep the illusion going by setting up the device
> > > > > start and end blocks accordingly. If you have, say, 32 physical
> > > > > blocks numbered from 0 to 31, then just set up startBlock=1 and
> > > > > endBlock =32;
> > > > >
> > > > > There is no chunk or block info imprinted in the actual YAFFS data,
> > > > > so no changes are needed to mkyaffsimage etc to make this work.
> > > >
> > > > Ugh, that sounds like a *total* hack.
> > >
> > > So what? It is very simple, works fine, is efficient and does not disrupt
> > > stable code.
> >
> > Yes it is a *simple* hack, but it builds on other "lies"(your words) in
> > the definition of YAFFS, i.e. that it *never* uses any data below the
> > first 'block' of flash.  For NAND devices where block sizes are small
> > compared to the number of blocks, then this is no problem.  In the case
> > where there are 4 8-bit flash parts, each with a 64k block size on a 32-
> > bit bus, you're talking about a 256K block size in 16M of memory which
> > is pretty darn large.
> >
> > > As it is, you're already pulling nastier hacks to emulate spare regions
> > > etc to make NOR look like NAND.
> >
> > That is because I want to use YAFFS on a NOR device because my hardware
> > has NOR devices.  Of course I have to pull a hack.  At least its a
> > *very* clean hack and hides itself from any layers above it.
> >
> > > >Is there anything wrong with
> > > > allowing dev->startBlock to be zero instead of 1?
> > >
> > > This would not work for various things.
> > > For example when YAFFS goes to look for the chunkId of the chunk in chunk
> > > zero, the search code  would report "it is in chunk zero" which the rest
> > > of yaffs interprets as "it does not exist". So anything you write to
> > > chunk zero is written into a black hole.
> >
> > Can you give me a pointer to function/line where this test is?  Perhaps
> > fixing YAFFS to handle block zero is better than forcing a 'hack' at a
> > lower layer.
> 
> 
> The main offendiung function is yaffs_FindChunkInGroup() that looks at 
> theChunk. If theChunk was passed in as zero, then it means "chunk does not 
> exist".

Hmm, I don't see yaffs_FindChunkInGroup in yaffs_guts.c, is that a
YAFFS2 function?  I see yaffs_FindChunkInFile() in yaffs_guts.c; is that
what you're referring to?

> >
> > > > I *could* do the remap in the access functions, but it *assumes some
> > > > magical offset that requires a comment of:
> > > >
> > > > 	/* This is a total hack since YAFFS refuses to use block zero */
> > >
> > > Hack, maybe. Total hack, I think, is stating it a bit strongly.
> >
> > Yes, "total" hack was a bit strong, but after using mkyaffsimage to
> > create a image that I put into flash, and finding that I couldn't find
> > some partial bits of it(and a bunch of objects end up in lost+found that
> > can't even be opened), I may have expressed my frustration a bit strong.
> > You can see where that comes from when 85% of the flash image is visible
> > and usable...
> >
> > > Any offset is fine so long as there is no logical (ie from YAFFS's point
> > > of view) chunk zero any more. Just using a one block offset is the
> > > simplest case. Counting from one instead of zero is hardly going to make
> > > most programmers spill their cup of tea.
> >
> > No, they won't spill their tea, but if you have 16MB of flash hooked up
> > that the minimum erase size is 256K, then that one lost block is
> > *really* aggrevating, especially since yaffs_mtdif.c doesn't mention
> > that block zero is ignored(and an offset should be applied).
> 
> This is mentioned in yaffs_fs.c.

Where?  a search of 'zero' over yaffs_fs.c yeilds a comment before
yaffs_delete_inode, referring to the link count, and 'block 0' only
refers to the places where dev->startBlock is set.  There's no comment
as to *why* it can't be zero....

> The mkyaffsimage output is fine, so long as you write it into flash starting 
> at startBlock.

Yeah, I figured that out, finally.  You can imagine if you loaded the
YAFFS image at block zero, and ignored the first 256K of data from the
image, some *really* wierd stuff happens including root directory
objects that never show up, and a lost+found that contains objects that
have no contents(due to missing objects located in the first 256K).

> >
> > > You can just use fixed constants in your mapping: Offset by one block.
> > > Since dev->nChunksPerBlock holds the number of chunks per block that is
> > > all you need to offset by for the reading/writing of a block.
> > >
> > > This means, all up, you need to add three lines of code:
> > >
> > >     blockId--; // add to the erase function
> > >
> > >     chunkId -= dev->nChunksPerBlock; // add to read/write functions
> > >
> > > And change two lines where you set the start and end blocks.
> > >
> > > Changing five lines of code is hardly a large effort. Much less than,
> > > say, writing an email.
> >
> > Only after its taken a long day of yanking your hair out to understand
> > the problem.  No offense, but yaffs_fs.c, yaffs_mdtif.c and
> > mkyaffsimage.c don't match as they exist now, and the documentation
> > doesn't mention that the image created by mkyaffsimage.c should be
> > offset by a flash block so that it all works.  Anyone who's marched down
> > this path should be able to tell you that.
> 
> 
> The mkyaffsimage output is completely location independent, but it is 
> expected to be used with mkyaffs (which loads starting at block one). If you 
> take the image and load it into flash at block 1 you'll be OK.

I understand that mkyaffsimage produces a location independent image,
but it *does* matter where it it is placed such that YAFFS actually
accesses all of it.

> In what way don't they match? 
> 
> AFAIK,
> 1) yaffs_fs.c uses startBlock of 1
> 2) mkyaffs loads from block 1
> 3) yaffs_guts says block zero is not permitted.

I have a separate bootloader that knows how to erase/burn flash.  I'm
trying to use YAFFS as a *root* filesystem, so it has to be burned from
a boot-loader, and not from Linux, so mkyaffs doesn't help me here.  I'd
figure if mkyaffsimage is like mkfs.jffs2, it produces an image that can
be burned *directly* into flash without the need for a support program
(such as mkyaffs).  And if YAFFS is my root FS, then I have a catch-22
here if I require mkyaffs...

Documentation/yaffs-rootfs-howto.html has:

<H3>Creating a bootable yaffs partition:</H3>
<P>You can</P>
<P>1) Create the partition by mounting it from a running linux os and
copying the data there. The mkyaffs utility in the yaffs source
simply erases a NAND mtdblock device without removing bad block
data.</P>
<P>2) Make and download a filesystem image. The mkyaffsimage utility
that came with the sources will create a YAFFS block list in a file
from a root tree. This is a list of 512+16 byte blocks that need to
be placed (in any order) on a NAND device.</P>
<P>You will need to write code to copy these data blocks and add in
the block numbers in the oob areas.</P>

Nowhere does that mention that mkyaffs is *required* to load the image
created by mkyaffsimage into the NAND device so that it 'shuffles' the
image past the first block.  It describes mkyaffs as 'simply erases a
NAND mtdblock device'.  That can be accomplished by most boot-loaders.

> <snip>
> 
> > Tell me where the code is that assumes block zero is "does not exist",
> > and I'll be glad to create/test(using LTP fs tests) a patch so the lower
> > layer is clean with regard to the usable start of flash...
> 
> The main offender is how the tnodes work and FindChunkInFile which assumes 
> that the chunk does not exist if the chunkId is zero.

I'll start looking there to see how to handle a block being zero.  I'm
assuming that an objectId of zero is really bad...

-- 
Peter Barada <Peter.B@LogicPD.com>