Re: [Yaffs] yaffs Digest, Vol 59, Issue 1

Top Page
This message is part of the following thread:
Mthe complete thread tree sorted by date
 
 
Attachments:
Message as email
+ (text/plain)
Delete this message
Reply to this message
Author: Josh Gelinske
Date:  
To: yaffs@lists.aleph1.co.uk
Subject: Re: [Yaffs] yaffs Digest, Vol 59, Issue 1
I appreciate the feedback, thank you. It also gives me a couple new ideas to maybe make what I am doing work.

What if I keep a list of sequence numbers for blocks that have been modified in some way since the last checkpoint and flush that at shutdown? I assume I would need to implement a new type of object / structure to do so. Then at boot I would scan anything in that list of sequence numbers and fix up the state from the last checkpoint.

Another idea is to add a checkpoint "transaction number" to the blockinfo. This number would increment every time I successfully write a checkpoint. Then at boot I would read in the checkpoint and scan the blockinfo to build a list of blocks with a "transaction number" greater than what the checkpoint has. I would then do the scan and fixup on those. This seems like less effort since I would not need to introduce a new object or type to store the data but tag on to some existing structures.


-----Original Message-----
From: [mailto:yaffs-bounces@lists.aleph1.co.uk] On Behalf Of
Sent: Thursday, April 01, 2010 6:00 AM
To:
Subject: yaffs Digest, Vol 59, Issue 1 

Send yaffs mailing list submissions to
    


To subscribe or unsubscribe via the World Wide Web, visit
    http://lists.aleph1.co.uk/cgi-bin/mailman/listinfo/yaffs
or, via email, send a message with subject or body 'help' to
    


You can reach the person managing the list at
    


When replying, please edit your Subject line so it is more specific
than "Re: Contents of yaffs digest..."


Today's Topics:

1. Periodic Checkpointing (Josh Gelinske)
2. Re: Periodic Checkpointing (Charles Manning)
3. Re: Periodic Checkpointing (James Kehl)
4. Re: Periodic Checkpointing (Charles Manning)
5. Re: Periodic Checkpointing (Ross Younger)


---------------------------------------------------------------------- 

Message: 1
Date: Wed, 31 Mar 2010 10:00:36 -0500
From: Josh Gelinske <>
Subject: [Yaffs] Periodic Checkpointing
To: "" <>
Message-ID:
    <>
Content-Type: text/plain; charset="us-ascii"


I have two large (4GB) NANDs in my system and I want to try implementing a periodic checkpoint mechanism to help speed up boot. Given the heavy amount of IO and the amount of state the checkpoint snapshots for these NANDS I cannot guarantee I can get a clean unmount (more like I can guarantee I will get an unclean shutdown).

My current thought is to add a kernel thread that will checkpoint every 30 seconds to 1 minute. Each time I "auto" checkpoint I would remove the older checkpoint once the new one is written. Of course given the timeing of shutdown can't be guaranteed this means I need to handle cleaning up that old checkpoint a boot as well in case it did not get erased. I have this much working and it improves boot time but of course there are issues.

The part I am missing and am unsure the best way to implement is the scan that I need to do for any changes after my checkpoint. Currently any modifications after my auto checkpoint are lost and this can cause corruption of existing data etc. Any ideas on the best way to do this? Am I completely going down the wrong path here?

Regards,

JG



JOSH GELINSKE
FIRMWARE ENGINEERING MANAGER

Appareo Systems, LLC
1810 NDSU Research Circle N
Fargo, ND 58102
P: (701) 356-2200 Ext 251
F: (701) 356-3157
http://www.appareo.com
mailto:jgelinske@appareo.com

{*}

NOTICE: This message (including attachments) is covered by the Electronic Communication Privacy Act, 18 U.S.C. sections 2510-2521, is CONFIDENTIAL and may also be protected by ATTORNEY-CLIENT OR OTHER PRIVILEGE. If you believe that it has been sent to you in error, do not read it. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error and then delete it.


{-}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.aleph1.co.uk/lurker/list/yaffs.html/attachments/20100331/79434ec0/attachment.html>

------------------------------

Message: 2
Date: Thu, 1 Apr 2010 13:04:24 +1300
From: Charles Manning <>
Subject: Re: [Yaffs] Periodic Checkpointing
To:
Message-ID: <>
Content-Type: text/plain; charset="utf-8"

On Thursday 01 April 2010 04:00:36 Josh Gelinske wrote:
> I have two large (4GB) NANDs in my system and I want to try implementing a
> periodic checkpoint mechanism to help speed up boot. Given the heavy amount
> of IO and the amount of state the checkpoint snapshots for these NANDS I
> cannot guarantee I can get a clean unmount (more like I can guarantee I
> will get an unclean shutdown).
>
> My current thought is to add a kernel thread that will checkpoint every 30
> seconds to 1 minute. Each time I "auto" checkpoint I would remove the older
> checkpoint once the new one is written. Of course given the timeing of
> shutdown can't be guaranteed this means I need to handle cleaning up that
> old checkpoint a boot as well in case it did not get erased. I have this
> much working and it improves boot time but of course there are issues.
>
> The part I am missing and am unsure the best way to implement is the scan
> that I need to do for any changes after my checkpoint. Currently any
> modifications after my auto checkpoint are lost and this can cause
> corruption of existing data etc. Any ideas on the best way to do this? Am I
> completely going down the wrong path here?

Hi Josh

One of the things I love about answering emails like this is that it prompts
new thinking and just responding has helped to seed a couple of ideas.

I've considered this a few times and have been thinking around what is
required to make an incremental checkpoint. There are some headaches...

The checkpoint holds quite a bit of cross-referenced state that would need to
be fixed up. For example:
* Writing a new chunk into a file will change the file chunk list, file info
and the block info (chunk use bitmaps, block state...).
* Doing a garbage collection might change many file chunk lists, block states,
chunk lists etc.
... and so on...

It would be fairly easy if all changes were incremental (ie just adding chunks
and no deleting). For example if we knew the checkpoint reflected state for
state up to the allocation of block sequence 5000 then saw that there's been
a block added with sequence 5001, we could just restore the checkpoint and do
a scan on block 5001 and apply those changes.

It gets a lot trickier if state has been changed and we then lose track of
those changes. For example, garbage collections, file deletions, etc.

I've started fiddling around with some ideas wrt checkpoint "patches". These
would be written to the end of the checkpoint and tell a post-scanner that it
needs to scan some blocks to fix up the state. Benefits are that a patch
would be very fast to write (quick to write as part of a shutdown handler,
maybe, or maybe written every time a block is deleted). Then every minute or
so the fresh checkpt could be written. This might get quite complicated... I
don't like complicated because it makes a breeding ground for bugs.

Another approach that I keep mentioning, and should really get around to
doing, is implementing "block summaries". This would use the last chunk in
each block to hold all the tags for chunks in the block. That makes scanning
way, way, faster meaning that the fallback for handling the case when there
is no current checkpoint is still approaches checkpoint speed.

These are not mutually exclusive and can work together. ie.
Use checkpoint if we can.
If not, scan, using block summaries for blocks that have them.

-- Charles






------------------------------ 

Message: 3
Date: Thu, 1 Apr 2010 10:57:03 +1000
From: "James Kehl" <>
Subject: Re: [Yaffs] Periodic Checkpointing
To: "Charles Manning" <>,
    <>
Message-ID:
    <>
Content-Type: text/plain;    charset="us-ascii"


> -----Original Message-----
> From: [mailto:yaffs-
> ] On Behalf Of Charles Manning
> Sent: Thursday, 1 April 2010 10:04 AM
> To:
> Subject: Re: [Yaffs] Periodic Checkpointing
>
...
> Another approach that I keep mentioning, and should really get around
to
> doing, is implementing "block summaries". This would use the last
chunk in
> each block to hold all the tags for chunks in the block. That makes
> scanning
> way, way, faster meaning that the fallback for handling the case when
> there
> is no current checkpoint is still approaches checkpoint speed.
>

Wouldn't this only be a gain for the inband-tags case, or when the MTD
driver can't read out a page's tags without reading out the entire page
as well (as some hardware ECC forces you to do)?

It would be interesting if this also let you save chunk
deletion/obsoletion flags - I heard someone pondering an
"incremental/background mount" idea once, where more files would become
available as the scan progressed. (Of course, any readdir() or access to
a non-existent file would have to block until the scan finished... but
with a usual layout, it might get init started a couple of seconds
earlier.) The biggest catch would be making sure old data (old
filenames, deleted files, old contents) weren't temporarily resurrected.

J




------------------------------

Message: 4
Date: Thu, 1 Apr 2010 14:59:42 +1300
From: Charles Manning <>
Subject: Re: [Yaffs] Periodic Checkpointing
To:
Message-ID: <>
Content-Type: text/plain; charset="iso-8859-1"

On Thursday 01 April 2010 13:57:03 James Kehl wrote:
> > -----Original Message-----
> > From: [mailto:yaffs-
> > ] On Behalf Of Charles Manning
> > Sent: Thursday, 1 April 2010 10:04 AM
> > To:
> > Subject: Re: [Yaffs] Periodic Checkpointing
>
> ...
>
> > Another approach that I keep mentioning, and should really get around
>
> to
>
> > doing, is implementing "block summaries". This would use the last
>
> chunk in
>
> > each block to hold all the tags for chunks in the block. That makes
> > scanning
> > way, way, faster meaning that the fallback for handling the case when
> > there
> > is no current checkpoint is still approaches checkpoint speed.
>
> Wouldn't this only be a gain for the inband-tags case, or when the MTD
> driver can't read out a page's tags without reading out the entire page
> as well (as some hardware ECC forces you to do)?

This should be beneficial all the time for the following reasons:
*The actual NAND page read costs the same time regardless of the number of
bytes being fetched from the page. Thus you'd be doing one of these rather
than 64. At, say, 20us per page read that's over 1ms of read time saved per
block.
* The summary would also reflect chunk deletions that happened before the
block was filled. Thus should speed the scanning somewhat for some use cases.

>
> It would be interesting if this also let you save chunk
> deletion/obsoletion flags - I heard someone pondering an
> "incremental/background mount" idea once, where more files would become
> available as the scan progressed.
> > (Of course, any readdir() or access to
> a non-existent file would have to block until the scan finished... but
> with a usual layout, it might get init started a couple of seconds
> earlier.) The biggest catch would be making sure old data (old
> filenames, deleted files, old contents) weren't temporarily resurrected.

In yaffs2 mode there are no problems with a scan building up obsolete data
because yaffs2 mode scans backwards so we always know that some data or a
file etc has been obsolteted before we stuff it into the tree. Of more
concern is incomplete data. If the directories and the files are only
partially populated then you can't tell the difference between a non-existent
directory entry and one that is still coming, or a sparse file and one that
is only partially populated with more data on the way.

There is a far better way to handle this: Partition off system boot files.
Even make them read-only and remount to rw to do mods then remount ro again.




------------------------------

Message: 5
Date: Thu, 01 Apr 2010 09:51:20 +0100
From: Ross Younger <>
Subject: Re: [Yaffs] Periodic Checkpointing
To:
Message-ID: <>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

James Kehl wrote:
> [block summaries]
> Wouldn't this only be a gain for the inband-tags case, or when the MTD
> driver can't read out a page's tags without reading out the entire page
> as well (as some hardware ECC forces you to do)?

It is true in my experience to say that tags mode is deathly slow - I
don't touch it if I can possibly avoid doing so.

OTOH there might well be an improvement to be had in inband tags mode if
you enable yaffs's own ECC on the tags and teach it to make a partial
page read of just the tags area. However I'm concentrating on large page
devices for the time being, so don't have any time to look further into
this.


Ross 

-- 
eCosCentric Ltd, Barnwell House, Barnwell Drive, Cambridge CB5 8UU, UK
Registered in England no. 4422071.                 www.ecoscentric.com




------------------------------

_______________________________________________
yaffs mailing list

http://lists.aleph1.co.uk/cgi-bin/mailman/listinfo/yaffs


End of yaffs Digest, Vol 59, Issue 1
************************************


This message was posted to the following mailing lists:
YAFFS
Mailing List Info | Nearby Messages		<-Re: [Yaffs] Periodic Checkpointing[Yaffs] Yaffs behaviour on mtd read failure->
Stoneboat Mailing List Archive administrated by JennyLurker (version 2.3)