Hi all I would like to let the vitriol of the last few week or so slide, but a lot of it contained some technical content that was wrong and thus I'd like to straighten out a few points on YAFFS2 and AUTOPLACE, hopefully in a reasonably coherent fashion. Three main assertions that have been made are wrong: 1) mtd interfaces are "golden" and cannot be changed. Thus YAFFS must go through any required tricks to make a working solution. 2) That it is both possible and wise to attempt this. 3) I have not a clue what AUTOPLACE is all about and how it should work. Below I make some comments on what Thomas has done. I am not levelling blame at him. He is a very busy man and works to a set of priorities that are not always what everyone else wants (besides, it is all open source so someone else could also jump in and fix ;-)). I have the highest regard for Thomas' knowledge and work. A POTTED HISTORY OF THE AUTOPLACE NAND INTERFACE Those that have been around YAFFS a long time will remember the original integration problems because YAFFS wanted to see things that mtd was not making available, in particular the oob info. To get this working, the first releases of YAFFS needed mtd patches to work. At that time, mtd nand support was very early days and soon the patch was no longer required as the mtd layer came up to speed (damn good effort from Thomas). YAFFS1 is designed for Smartmedia data layout and thus there are the hard-wired yaffs_Spare structures etc. Thus, when more, and different flash layouts were introduces, things like the oob_sel were introduced to allow the generic nand code to work with various byte-layouts and ECC mechanisms. oob_sel has also changed a few rimes. YAFFS1 suffered some minor traumas due to this, but nothing too bad. When I started designing YAFFS2, about 2.5 years ago now, one of the main goals was to be able to support a much wider range of NAND devices and work with different hardware ecc etc. So, one of the first areas that was designed was the NAND interface. Clearly fixed binary structures would not work properly and a level of abstraction was required. This drove the move from yaffs_Spare to yaffs_ExtendedTags and the packed tags mechanisms. As things slowly unfolded, I realised that it would be a better thing to get buy-in from Thomas sooner, rather than layer, as to how to progress 2k and other support in mtd. It is far better to get a sensible interface in place earlier than try to retro-fit things later. As YAFFS is a customer to those interfaces, it made sense to get involved and discuss things. So in late 2003, Thomas and I started some discussions on this subject. [Much of the discussion was on IRC, and some was email. I deleted some of my old email, but I still have some stuff I sent in October 2003]. My starting point was a more abstract NAND interface that did not require any knowledge of actual byte placement. [See below for a technical outline of the rationale]. Thomas, being a very knowledgeable fellow, also brought along a bunch ideas - many of them on the same wavelength. The results of all this were a definition of the functionality, but not an absolute function call definition, of AUTOPLACE and abstract bad block handling. I then continued with YAFFS2 development, outside of Linux, and a YAFFS2 prototype was being stress tested by Christmas 2003. In January it was being stress tested on large arrays of 2k page NAND. Note that this was a stripped YAFFS1 and did not support simultaneous yaffs1/2 functionality or yaffs1 compatability. In approx April 2004, Thomas started working on the AUTOPLACE and bad block handling. IIRC, this was all in place by the end of May. I then set about retooling yaffs2 to support both yaffs2 and yaffs1 formats through a backward compatability layer (tags compatability) - in essence a fusion of both the YAFFS1 and YAFFS2 prototype. This effort got somewhat delayed by me taking some time off from YAFFS for personal reasons. When I started checking out the YAFFS2<->mtd interfacing I used a small ram emulation driver that I hacked up quickly. This conformed to the interface that Thomas and I agreed on. Then YAFFS2 was released on the world. A few people picked it up and started playing. There were a few things that were patched that fixed some of the fusion problems (eg. Nick Ban'es patches for compatability mode). Unfortunately it takes time before people switch over and start testing with the 2k page devices. Some of the bugs discovered were due to some problems in the mtd not doing the AUTOPLACE properly (!!shock! horror! Thomas is human!!). JFFS2 had not shown this because it does not rely on oob to the same extent. This was discussed on the list sometime in May 2005. Some aspects of this were corrected, but it is not yet fully sound. Since then, the AUTOPLACE thing has been somewhat of an open wound, awaiting resolution. During that time, pragmatic people have worked around the problem, however it has not been cured yet. The good news though is that a fine fellow by the name of Vitaly Wool has started look at at some issues and clearly identifies the problem (see http://lists.infradead.org/pipermail/linux-mtd/2005-September/013949.html). I therefore hope for some resolution to this pretty soon. If Vitaly's suggestions come into being then the current code in YAFFS2 will work as it is with no modifications. So that is pretty much a history of the AUTOPLACE business. Things don't always progress the way one hopes, and looking back I'd have done some things a bit differently. As to these mtd interfaces being golden kernel interfaces: Not many people think so. Thomas doesn't. grep a whole Linux+yaffs source tree for read_oob and you'll only see references in mtd and yaffs. It is not like we're saying "change kalloc". There is no immediate solution, but it looks like the proper solution should be there soon. I see Vitaly has proposed some patches. Thomas has asked me to look at them and comment. -- CHarles [Technical side bar: Why the correct solution is to fix it in mtd] There are many NAND types and implementations of hardware ECC etc, which means there are a lot of different ways to to bad block management and many different byte layouts on NAND. For instance: 1) The default nand_base uses bytes 0 and 1 for bad block marking. 2) The HW_ECC for the S3C2410 uses these bytes for ECC. 3) Some, perhaps not designed yet, hardware might do something entirely different. Further, one NAND driver might be serving up data to many file systems, so it is unreasonable to have a file system's preferred binary layout This might even be impossible with some hardware anyway. One of the fundamental tenets of Computer Science is to use abstract interfaces to hide detail. It is this thinking which lead to AUTOPLACE and the abstract bad block handling interface. This is nothing new and examples of abstract interfaces abound. So the AUTOPLACE handling is there to provide a mechanism for abstracting away the physical location of oob bytes. It says "here are some bytes, save them whereever. When I ask for them back, then give them back. What every you do behind thescenes I don't want to know". With an abstract interface, we can get around a lot of problems quite neatly: 1) Changes in mtd don't mean changes to YAFFS, or other file systems. 2) Implementation of some funky handware can be done without implementing new fields in oob_sel structures and having to fiddle multiple bodies of code. Optimal handling of data (for example special hardware tricks can be exploited). 3) Change in one place, test in one place, then it should "just work". If we don't use an abstract interface then we get into a world of pain and we end up with code looking like http://www.aleph1.co.uk/pipermail/yaffs/2005q4/001581.html, which still does not handle all cases. This raises a bunch of problems including, but not limited to: 1) Replicated code. The algorithms are already in mtd so we should rather use them there. 2) mtd interaction. mtd changes. oobsel will most likely change again (it is now in about its third or fourth iteration). Don't want to have to change YAFFS and put in conditional code for even more changes. 2) Maintenance woes: We want to test YAFFS *once* and not have to test it against a zillion different hardware types etc and handle all the associated patches etc. When a problem is found, strict adherence to interfaces helps isolate problems. 3) If someone makes some new hardware/drivers that do not use oobsel, then we don't want to have to have to expose more detail to the outside world. 4) It is insane to use an abstract AUTOPLACE interface for some accesses, and not for others. It is important to use abstract interfaces consistently otherwise why have them at all? {Analogy: When you write data to a serial port driver, you use some relatively abstract interface like write_byte(), you don't do something like outp(dev->uart->tx,b).] -- Charles