Re: [Yaffs] Weirndess testing YAFFS2 with large files - md5s…

Hello Peter

On Tuesday 26 January 2010 09:39:46 Peter Barada wrote:
> I've run into a problem using the latest YAFFS code on linux-2.6.28-rc8
> using today's YAFFS CVS code.

Does the test work with older yaffs?

>
> First off I had to modify the code in yaffs_write_begin to only call
> grab_cache_pages_write_begin if the kernel version is newer than 2.6.28
> since 2.6.28-rc8 does *not* have __grab_cache_page whereas 2.6.28 proper
> does:
>

> #if LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 28)
>     pg = grab_cache_page_write_begin(mapping, index, flags);
> #else
>     pg = __grab_cache_page(mapping, index);
> #endif

That is an unfortunate problem due to there being no way to conditionally
compile against rc and other sub-version markers. I originally had this the
way you have it now, but changed it.

>
> Second I added "dev->nPageWrites++;" to
> nandmtd2_ReadChunkWithTagsFromNAND and "dev->nPageReads++;" to
> nandmtd2_WriteChunkWithTagsToNAND to track the number of page
> read/writes.

You should not have to. Those were moved out and placed in yaffs_nand.c

>
> With this code, I'm seeing 30MB files that are created have mismatching
> checksums while running the attached test script. The output from the
> test looks like:
>
> OMAP-35x# . /media/mmcblk0p1/x
> Create 30M file and get
> md5sum
> 30720+0 records
> in
> 30720+0 records
> out
> 5b04790304a4221f1016a8c310da4746
> somefile.1
> **>> Block 710 needs
> retiring
> **>> yaffs write required 2
> attempts
> **>> Block 710
> retired
> Block 710 is in state 9 after gc, should be
> erased
> Calculate md5sums for copied
> files
> 5b04790304a4221f1016a8c310da4746
> somefile.1
> 5b04790304a4221f1016a8c310da4746
> somefile.2
> 5b04790304a4221f1016a8c310da4746
> somefile.3
> 8c8d5a7974d0b9da747bc59edd1991f6
> somefile.4
> execute sync and recalculate
> md5sums
> save exit: isCheckpointed
> 1
> 5b04790304a4221f1016a8c310da4746
> somefile.1
> 5b04790304a4221f1016a8c310da4746
> somefile.2
> 5b04790304a4221f1016a8c310da4746
> somefile.3
> 5b04790304a4221f1016a8c310da4746
> somefile.4
> Delete one of the
> files
> 5b04790304a4221f1016a8c310da4746
> somefile.1
> 5b04790304a4221f1016a8c310da4746
> somefile.3
> 3cb7668eb7760202d96970a6a9a3361f
> somefile.4
> recopy the deleted
> file
> f6dba6d5af7a7a89481da1849035a417
> somefile.1
> 5b04790304a4221f1016a8c310da4746
> somefile.3
> 5b04790304a4221f1016a8c310da4746
> somefile.4
> f6dba6d5af7a7a89481da1849035a417
> somefile.7
> Creating test folder and some junk files in that
> folder
> 1+0 records
> in
> 1+0 records
> out
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.1
> md5sums of all files in test
> folder
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.1
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.2
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.3
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.4
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.5
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.6
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.7
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.8
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.9
> execute sync and recalculate
> md5sums
> save exit: isCheckpointed
> 1
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.1
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.2
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.3
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.4
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.5
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.6
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.7
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.8
> ae1028b8d6aef86d020c9edfae29ca3d
> junk.9
> Remove some files and recreate
> them
> Calculate md5sums for 30M files
> again
> f6dba6d5af7a7a89481da1849035a417
> somefile.1
> 5b04790304a4221f1016a8c310da4746
> somefile.3
> 5b04790304a4221f1016a8c310da4746
> somefile.4
> 1fee3f481bfa5cf3403efe9e481a0374
> somefile.7
> execute sync and recalculate
> md5sums
> save exit: isCheckpointed
> 1
> a7b6ccfa31115aa75a0fdca07073293d
> somefile.1
> 5b04790304a4221f1016a8c310da4746
> somefile.3
> 1abb3d578e2d129341df26916090b869
> somefile.4
> f6dba6d5af7a7a89481da1849035a417
> somefile.7
> OMAP-35x#
>
> In the output, note that the md5sum of "somefile.*" should all match.
>
> Anyone seen anything like this before? Test attached.

I just ran the test on both 2.6.24-xxx and 2.6.31-xxx using nandsim on a PC
and had no problems. Here's one run:

root@linux-dual-head:/mnt# ~charles/Dropbox/yaffs-30M-test
Create 30M file and get md5sum
30720+0 records in
30720+0 records out
31457280 bytes (31 MB) copied, 7.72198 s, 4.1 MB/s
dc7fd1b9553217a9a1becbb101271eab somefile.1
Calculate md5sums for copied files
dc7fd1b9553217a9a1becbb101271eab somefile.1
dc7fd1b9553217a9a1becbb101271eab somefile.2
dc7fd1b9553217a9a1becbb101271eab somefile.3
dc7fd1b9553217a9a1becbb101271eab somefile.4
execute sync and recalculate md5sums
dc7fd1b9553217a9a1becbb101271eab somefile.1
dc7fd1b9553217a9a1becbb101271eab somefile.2
dc7fd1b9553217a9a1becbb101271eab somefile.3
dc7fd1b9553217a9a1becbb101271eab somefile.4
Delete one of the files
dc7fd1b9553217a9a1becbb101271eab somefile.1
dc7fd1b9553217a9a1becbb101271eab somefile.3
dc7fd1b9553217a9a1becbb101271eab somefile.4
recopy the deleted file
dc7fd1b9553217a9a1becbb101271eab somefile.1
dc7fd1b9553217a9a1becbb101271eab somefile.3
dc7fd1b9553217a9a1becbb101271eab somefile.4
dc7fd1b9553217a9a1becbb101271eab somefile.7
Creating test folder and some junk files in that folder
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000442124 s, 2.3 MB/s
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.1
md5sums of all files in test folder
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.1
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.2
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.3
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.4
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.5
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.6
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.7
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.8
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.9
execute sync and recalculate md5sums
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.1
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.2
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.3
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.4
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.5
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.6
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.7
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.8
0bc0c6e9588ee2bf6c89463208c5a0e9 junk.9
Remove some files and recreate them
Calculate md5sums for 30M files again
dc7fd1b9553217a9a1becbb101271eab somefile.1
dc7fd1b9553217a9a1becbb101271eab somefile.3
dc7fd1b9553217a9a1becbb101271eab somefile.4
dc7fd1b9553217a9a1becbb101271eab somefile.7
execute sync and recalculate md5sums
dc7fd1b9553217a9a1becbb101271eab somefile.1
dc7fd1b9553217a9a1becbb101271eab somefile.3
dc7fd1b9553217a9a1becbb101271eab somefile.4
dc7fd1b9553217a9a1becbb101271eab somefile.7


Perhaps the retirement of the blocks indicates that some data was being
corrupted.

-- Charles