Greetings all,
I have been using yaffs for a few years now (with no issues) and have
recently been experiencing a bit of a mystery. I am using openwrt and I
should probably be posing this question/issue over there, but I have a
feeling you guys will have some better insight into this particular
problem. (note: the files are written directly to flash, there is no
overlay filesystem like some OpenWRT images)
The basic symptom is that the disk is reporting full, but there are not
enough files to legitimately take up that much space. These systems were
previously working, and are still running (in the last few months they have
been constrained by "no free space").
When I do "df -h", it reports the system is full, but when I do a sort of
crude totaling of file sizes (du -hcs /), it is reporting significantly
less usage.
root@errored_system:/root# du -hcs /
16.3M /
*16.3M* total
root@errored_system:/root# df -h
Filesystem Size Used Available Use% Mounted on
rootfs 58.0M *58.0M* 8.0K 100% /
/dev/root 58.0M 58.0M 8.0K 100% /
tmpfs 30.3M 444.0K 29.8M 1% /tmp
tmpfs 512.0K 0 512.0K 0% /dev
(~41.7MB unaccounted for.... thats more than can be justified by blocks
that are not full)
I have looked for files that might be sucking up this much space, but have
not found any (big files, or lots of small files). This leads me to believe
there is something amiss with the file system and why I am seeking your
insight.
I don't know much about debugging yaffs specifically, but this seems like a
good clue:
root@errored_system:/root# echo "-all+gc">/proc/yaffs
root@errored_system:/root# dmesg | tail -f
[ 3419.220000] yaffs_block_became_dirty block 3048 state 8
[ 3419.220000] GC Selected block 3437 with 1 free, prioritised:0
[ 3419.220000] yaffs: GC erasedBlocks 5 aggressive 1
[ 3419.260000] yaffs_block_became_dirty block 3437 state 8
[ 3419.270000] GC Selected block 2201 with 1 free, prioritised:0
[ 3419.270000] yaffs: GC erasedBlocks 5 aggressive 1
[ 3419.310000] yaffs_block_became_dirty block 2201 state 8
[ 3419.310000] GC Selected block 2231 with 1 free, prioritised:0
[ 3419.310000] yaffs: GC erasedBlocks 5 aggressive 1
[ 3419.370000] yaffs_block_became_dirty block 2231 state 8
These "yaffs_block_became_dirty block" records rapidly fill up the kernel
log... (on the good systems, I do not get these messages, at least not in
rapid succession)
Heres the interesting thing: they are all repeating on the same few blocks.
(2201, 2357, 2819, 3048 & 3437... same story on the other error'd systems,
only different block numbers) So my hunch is that something is preventing
these blocks from getting GC'd and are somehow backlogging the other blocks
that need freeing. Is that even possible? I am able to delete legitimate
files and regain their space, but that doesn't really address the issue of
the mystery disk usage. It does tell me that the GC is working at least in
part.
Incidentally, I cross-referenced the yaffs_block_became_dirty block against
the "bad eraseblocks" and "blocks marked bad" as reported on startup and
they were different (i.e. the blocks in the above GC messages were not
"known bad blocks"
Is there any way to forcibly verify the integrity of the data blocks in
yaffs and free up this mystery space? In other words, I have remote access
to these systems, but re-flashing isn't exactly an option for me as they
are geographically distributed (i.e. data collectors). Can I somehow force
those block numbers to be marked as bad? (I'd imagine that wouldn't be
wise, even if I could)
Some more (hopefully) useful information:
I am not writing much to flash. About once an hour I update a couple of
values in a sqlite database. I dont know if the update operation is the
culprit (since its behaving on other systems), but its the only source of
disk IO I can think of.
There are not many reported bad blocks (5-20 depending on which system I'm
looking at)
The board manufacturer apparently has changed the NAND chip somewhere along
the line. I am experiencing the issue on both types (so I'm fairly certain
its not exclusively hardware related)
One system:
[ 2.520000] NAND device: Manufacturer ID: 0xad, Chip ID: 0x76 (Hynix
NAND 64MiB 3,3V 8-bit)
Another system:
[ 2.510000] NAND device: Manufacturer ID: 0xec, Chip ID: 0x76 (Samsung
NAND 64MiB 3,3V 8-bit)
Here is the /proc/yaffs from a good system (one that isn't full)
root@good_system:/root# cat /proc/yaffs
Multi-version YAFFS built:May 25 2012 01:48:02
Device 0 "rootfs"
start_block.......... 0
end_block............ 3711
total_bytes_per_chunk 512
use_nand_ecc......... 1
no_tags_ecc.......... 0
is_yaffs2............ 0
inband_tags.......... 0
empty_lost_n_found... 0
disable_lazy_load.... 0
refresh_period....... 500
n_caches............. 10
n_reserved_blocks.... 5
always_check_erased.. 0
data_bytes_per_chunk. 512
chunk_grp_bits....... 0
chunk_grp_size....... 1
n_erased_blocks...... 294
blocks_in_checkpt.... 0
n_tnodes............. 7901
n_obj................ 5653
n_free_chunks........ 55483
n_page_writes........ 597395
n_page_reads......... 906712
n_erasures........... 11524
n_gc_copies.......... 22402
all_gcs.............. 32575
passive_gc_count..... 32575
oldest_dirty_gc_count 0
n_gc_blocks.......... 11524
bg_gcs............... 0
n_retired_writes..... 0
nRetireBlocks........ 0
n_ecc_fixed.......... 0
n_ecc_unfixed........ 0
n_tags_ecc_fixed..... 0
n_tags_ecc_unfixed... 0
cache_hits........... 100140
n_deleted_files...... 37
n_unlinked_files..... 22541
refresh_count........ 0
n_bg_deletions....... 0
And heres one from a "disk full" system:
root@errored_system:/root# cat /proc/yaffs
Multi-version YAFFS built:May 25 2012 01:48:02
Device 0 "rootfs"
start_block.......... 0
end_block............ 3711
total_bytes_per_chunk 512
use_nand_ecc......... 1
no_tags_ecc.......... 0
is_yaffs2............ 0
inband_tags.......... 0
empty_lost_n_found... 0
disable_lazy_load.... 0
refresh_period....... 500
n_caches............. 10
n_reserved_blocks.... 5
always_check_erased.. 0
data_bytes_per_chunk. 512
chunk_grp_bits....... 0
chunk_grp_size....... 1
n_erased_blocks...... 5
blocks_in_checkpt.... 0
n_tnodes............. 23388
n_obj................ 19836
n_free_chunks........ 206
n_page_writes........ 2912051
n_page_reads......... 2907834
n_erasures........... 88768
n_gc_copies.......... 2751776
all_gcs.............. 88768
passive_gc_count..... 0
oldest_dirty_gc_count 0
n_gc_blocks.......... 88768
bg_gcs............... 0
n_retired_writes..... 0
nRetireBlocks........ 0
n_ecc_fixed.......... 1
n_ecc_unfixed........ 0
n_tags_ecc_fixed..... 0
n_tags_ecc_unfixed... 0
cache_hits........... 1851
n_deleted_files...... 2
n_unlinked_files..... 35064
refresh_count........ 0
n_bg_deletions....... 0
Thanks,
Walter
(text/plain)