Skip to content

forest-cli archive export --epoch=X #3159

Closed
#3169
Closed
forest-cli archive export --epoch=X#3159
#3169
Assignees
ruseinov

Description

@lemmih
lemmih
opened on Jul 10, 2023

A Forest node can generate up-to-date snapshot files. However, we often want to take large snapshots and trim them to a specific epoch. This can be done entirely without network access.

The forest-cli archive export command takes a snapshot file as input and yields a smaller snapshot file as output. For example, the input snapshot might cover epochs 1 to 5000, and the output snapshot could cover epochs 1000 to 2000.

Subtasks:

  • Take CAR file as input,
  • Index CAR file using car-backed Blockstore,
  • Get recent-roots and epoch from cli flags,
  • Use export function to generate snapshot data,
  • Compress with zst.

Error handling:

  • recent-roots too low (less than chain_finality),
  • recent-roots too high (greater than available roots),
  • epoch not in CAR input file,
  • input file may be malformed or corrupted.

Export function

forest/src/chain/store/chain_store.rs

Lines 528 to 535 in 4f0d75f

pub async fn export<W, D>(
&self,
tipset: &Tipset,
recent_roots: ChainEpoch,
writer: W,
compressed: bool,
skip_checksum: bool,
) -> Result<Option<digest::Output<D>>, Error>

Activity

  1. ruseinov
    self-assigned this
    on Jul 10, 2023

  2. lemmih commented on Jul 10, 2023

    @lemmih
    lemmih
    on Jul 10, 2023
    ContributorAuthor

    Testing can be done by downloading a calibnet snapshot and re-exporting it in various ways (different offsets, different number of tipsets).

  3. lemmih commented on Jul 10, 2023

    @lemmih
    lemmih
    on Jul 10, 2023
    ContributorAuthor

    Automated testing with the MemoryDB would also be nice.

  4. ruseinov commented on Jul 11, 2023

    @ruseinov
    ruseinov
    on Jul 11, 2023
    Contributor

    @lemmih Do we also want the checksum here?

  5. lemmih commented on Jul 11, 2023

    @lemmih
    lemmih
    on Jul 11, 2023
    ContributorAuthor

    @lemmih Do we also want the checksum here?

    We can leave out the checksum for now.

  6. ruseinov commented on Jul 11, 2023

    @ruseinov
    ruseinov
    on Jul 11, 2023
    Contributor

    One more question: do we really want to call a param recent-roots? I would probably do something like
    epoch-from, epoch-to and make sure that epoch-from is at least chain_finality` (defaulting to it too).

    We could come up with even better naming perhaps
    --epoch and --depth.

  7. lemmih commented on Jul 11, 2023

    @lemmih
    lemmih
    on Jul 11, 2023
    ContributorAuthor

    One more question: do we really want to call a param recent-roots? I would probably do something like epoch-from, epoch-to and make sure that epoch-from is at least chain_finality` (defaulting to it too).

    We could come up with even better naming perhaps --epoch and --depth.

    --epoch and --depth looks good.

  8. ruseinov
    changed the title [-]`forest-cli archive snapshot --epoch=X`[/-] [+]`forest-cli archive export --epoch=X`[/+] on Jul 12, 2023
  9. ruseinov
    closed this as completedin #3169on Jul 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions