bundle: filesystem metadata format

@stevvooe

@stevvooe and I caught-up in person about our digest discussion and the need for serialize file-system metadata. If you want to read my attempt it is found here: #5 (comment)

Problem: a rootfs for a container bundle sitting on-disk may not reflect the exact intended state of the bundle when it was copied to its current location. Possible causes might include: running on filesystems with varying levels of metadata support (nfs w/o xattrs), accidental property changes (chown -R), or purposeful changes (xattrs added to enforce local policies).

Obviously the files contents will be identical so that isn't a concern.

Solution: If we hope to create a stable digest of the bundle in the face of these likely scenarios we should store the intended filesystem metadata into a file itself. This can be done in a variety of ways and this issue is a place to discuss pros/cons. As a piece of prior-art @vbatts has implemented https://github.com/vbatts/tar-split and we have the linux package managers with tools to verify and restore filesystem metadata from a database with rpm -a --setperms and rpm -V.

@shykes

After some thought, we need a format that does the following:

A manifest enumerates files in a bundle.
1. Provides a content hash.
2. Provides a path.
3. Provides a file type.
4. Provides standard file mode.
5. Provides xattr.
6. Provides an extension mechanism.
  1. Geared towards multiple OS support.
  2. Not infinitely extendable but should be easy to add new field.
Bundle contents attributes can be reset to contents of file manifest.
1. Bundle is scanned and any differences from manifest are rectified.
2. Unames/Gnames/Uid/Gid can be mapped during "reset".
Bundle contents can be verified against manifest.
1. Content hash can be checked.
2. Attributes can be checked.
  1. Certain attributes can be checked against machine-local mapping (uid/gid, etc.).
3. Manifest can optionally be signed.

Meeting the above three use cases with this format puts us above the test for Cost problems with the ideas from #5. Requirement 2 above has a common use case, by avoiding placing unreasonable requirements on transports. Requirement 3 above gives us the functionality of #5 with the extra benefits of Requirement 2.

@shykes @philips @crosbymichael

@stevvooe

@stevvooe I am having a hard time parsing this sentence: "Meeting the above three use cases with this format puts us above the test for Cost problems with the ideas from #5."

I agree with all three needs overall despite my confusion above.

@philips

@philips That is a poor sentence where I've shoved in a lot of meaning.

What I'm saying is that, given the goals of #5 (cryptographic verification), the cost of scanning a bundle is not warranted. Given these new goals, the cost is warranted and it makes bundles portable over different transports. Basically, we have a solid reason for filesystem scanning that could also be used as a signable target.

@stevvooe

@stevvooe Ack. So the next step is a .proto file?

No better way to get started than with a straw man:

syntax = "proto3";

package ocf.bundle;

// BundleManifest specifies the entries in a container bundle, keyed and
// sorted by path.
message BundleManifest {

    message Entry {
        // path specifies the path from the bundle root
        string path = 1;

        // NOTE(stevvooe): Need to define clear precedence for user/group/uid/gid precedence.

        string user = 2;
        string group = 3;

        uint32 uid = 4;
        uint32 gid = 5;

        // mode defines the file mode and permissions. We've used the same
        // bit-packing from Go's os package,
        // http://golang.org/pkg/os/#FileMode, since they've done the work of
        // creating a cross-platform layout.
        uint32 mode = 6;

        // NOTE(stevvooe): Beyond here, we start defining type specific fields.

        // digest specifies the content digest of the target file. Only valid for
        // regular files. The strings are formatted as <alg>:<digest hex bytes>.
        // The digests are added in order of precedence favored by the 
        // generating party.
        repeated string digest = 7;

        // target defines the target of a hard or soft link, relative to the
        // bundle root.
        string target = 8;

        // specifies major and minor device numbers for charactor and block devices.
        string major = 9;
        string minor = 10;

        message XAttr {
            string name = 1;
            string value = 2;
        }

        // xattr provides storage for extended attributes for the target resource.
        repeated XAttr xattr = 11;

        // AlternateDataStream represents NTFS Alternate Data Streams for 
        // the targeted resource.
        message AlternateDataStream {
            string name = 1;
            bytes value = 2;
        }

        // ads stores one or more alternate data streams for the given resource.
        repeated AlternateDataStream ads = 12;
    }

    repeated Entry entries = 1;
}

Changes:

digest field is now repeated
Added description for AlternateDataStream type, formerly ADS

@stevvooe

@stevvooe looks pretty good. Two questions:

What is an ADS?
Should digest be repeated so we can deprecate old hashes and upgrade to new ones over time?

What is an ADS?

This is NTFS equivalent of extended attributes (sort of), known as "Alternate Data Streams". The semantics are slightly different, so I've pulled it out into a separate type. Notice the use of type bytes for the value, instead of string. I'd like to get some feedback from a Windows expert to see if this is sufficient.

Should digest be repeated so we can deprecate old hashes and upgrade to new ones over time?

In this case, I don't see why not.

I've updated the comment in-line.

@philips

@philips We may also want to define an exclusion operator to the manifest specification, since it operates at the bundle level.

enum Op {
    // EXCLUDE specifies that the matched files should be explicitly exlcuded
    // from the manifest. They may be still part of the bundle.
    EXCLUDE = 0;

    // INCLUDE specifies that the resource should be include in the manifest.
    // This has the act of "pinning" the resource. For example, if the resouce
    // is later matched by an exclude statement, it will still be included.
    INCLUDE = 1;
}

message PathSpec {
    Op operation = 1 [default=EXCLUDE];

    // path specifies a path relative to the bundle root. If the path is a
    // directory, the entire tree will be excluded.
    string path = 2;

    // pattern specifies a glob to match resources and apply the operation.
    string pattern = 3;
}

// path specifies the bundle paths covered by the manifest. Specifications are
// order by precedence. For a given path, only the first matching
// specification applies. During processing, it is important to fully process
// all resources, even if a directory is excluded, since child resources may
// first match an inclusion.
repeated PathSpec pathSpec;

Benefits:

Very explicit for recalculation
Allows us to catch files that may have showed up and don't belong

Cost:

More processing time
Subtle include/exclusion behavior to minimally specify file sets
Can contribute to instability of file -- equivalent manifests may have different specs

Another possibility is to allow this to be specified on the command line when first building the manifest. That doesn't allow us to catch "extra" files, but that may not be that important and likely doesn't warrant the extra complexity.

Okay, so after reading this, I think this makes a bit more sense to me . . . I think the area where I'm having trouble is understanding the scope vis a vis the goals...

I think what I've read so far is well thought out and reasoned, so props to everyone , heh. Forgive me ahead of time, but I wanted to share some thoughts. Discard if you don't think they are useful.

There seems a basic consensus -- in the limited info I've read on the container crypto goals --that everyone would probably be on board with the following ideas /in principle/ , certainly as options... These are labeled as the 'Basic Themes' below .. areas where there's agreement:

Basic Themes

Containers may need to be secured against A-MITM in-transit (integrity)
Containers may need to be secured against A-MITM during distribution (integrity)
We need a way of validating and identifying who is the author of a container (authentication, non-repudiation, integrity)
Containers do NOT need their own transport layer, since we are assuming TLS / OpenSSH will take care of that .
<--- Additional reasons go here --->

Just as general thought, first I'm going to put these here just as problems I have run into myself with design of cryptography... You guys may know this already but writing this down helps me organize my thoughts.

I'll be back with specifics tomorrow or Saturday.

Statements in general I've found true in crypto engineering. May be useful in this context.

Never reinvent the wheel in crypto -- this is the 'once upon a time' for systems that get owned
If the full description & implementation of yr system can withstand the scrutiny of a 16-year old Norwegian hacker named Jon , such that it cannot be cracked in 8 lines of Perl, then you are off to a decent start. (unlike DeCSS)
Always aim for zero-knowledge even though it's near impossible to do right... The line of thinking provides insight regardless.
Never put new / novel cryptography and/or standards directly into production -- if necessary , only do so if it's backed by scientific research and a rock-solid implementation
Don't do an implementation from scratch -- always use an open-source well-tested library like NaCl, PyCrypto, etc.
Side channel attacks are not theoretical (Heartbleed, WEP, Lucky13, etc). Leaky system borders are where side channels live.
Padding oracle attacks like Heartbleed from crappy solutions in SSL/TLS RFCs for how to authenticate packets... Do not repeat, heh.
Work at the highest level possible, in primitives/concepts rather than in methods
Use nonces, timestamps, & digital signatures wherever possible to avoid replay attacks
Correct forward operation is Compress, Encrypt, then MAC/Authenticate (never the opposite)... The correct reverse operation is MAC/Authenticate, Decrypt, then Decompress. ("Cryptographic Doom Principle" for side channels)
http://www.thoughtcrime.org/blog/the-cryptographic-doom-principle/
Study existing protocols... always adapt an existing format or protocol instead of writing one from scratch
Scope creep in crypto can be brutal, it goes on forever

- Define our threat model before we do anything else. Who are we trying to protect against?

What are we trying to protect (data, secrets, access, etc)? What is our tolerance in terms of a threat vs it's complexity/likelyhood?

I've got a series of specific questions in regards to the proposed standards here (which I think are pretty awesome )... but I'm tired at the moment -- I'll post the specific questions / comments tomorrow: I hope this helps in the time being . . . This is my take on how to think about crypto engineering.

General Questions:

What is our threat model? Who is the attacker? What are their capabilities/
What does a passive attack look like (if one is possible here)?
What form is an active attack? What is the weakest link in the existing thinking?
Do we need Digests? What threat do they address and how?
Do we need Digital Sigs / MACs? How are Digital Sigs keyed? What threat do they address?
Do we have a line of thinking for the key distribution model? WoT / CA etc. Here it's often there's no choice but that we must piggyback on an existing implementation like OpenPGP (like Ubuntu PPA)
Based on the threat model, what are the appropriate countermeasures (aka cryptogrpahic primitives)?
Do we need to support encryption as a use-case, or can we cut it with just hashes / digital sigs?

Specific Questions (assuming key distribution is solved, heh... like with OpenPGP):

Should we deploy Digital Signatures to secure 'single-file' compressed container images (like rkt does now)?
Do we know if containers will always (or at least almost all of the time) be distributed on the Internets as some sort of compressed single-file archive ( exact format is irrelevant)?
If containers are distributed as a single-file archive, and these are secured properly with digital sigs and/or GPG etc, why is it necessary (or beneficial) to secure the data-at-rest in an uncompressed container?
Has a container format been finalized or standardized beyond the json manifest? What is certain about the container archive format, and what is up in the air? What will be up to the implementation, and what is defined by the spec?
What are the advantages of securing data-at-rest inside the container vs simply securing a container archive image file? Do we gain security here? If so, what specific attack does this prevent?
What are the pitfalls in scope creep and/or other impacts of securing data beyond a single-file container archive?
Are there any compromise solutions that lie between these two ideas?
What is the threat model we are considering?
Is a threat where someone tricks me into downloading a malicious container?
Or does someone poison the ARP table at a starbucks while I am downloading the container image, such that I download their container image?
Are we protecting against a 'runaway app' inside a container? A malicious actor on the host system which has done privilege escalation? Industrial espionage by ceiling cat?
What sort of primitives do we need to achieve stated level of security?
What can we re-use to minimize the effort and make it easy (or certainly at least straightforward) for others to implement this standard?
What can we re-use to avoid implementation , testing, and possible exploit headaches?

BTW these questions arent meant to all have answers obviously. They are more like engineering food .

Anyway -- thanks, good work, and good luck gentleman. Looking forward to writing about the details tomorrow if time permits, as you all have some really good ideas here, and I'm sure you'll sort this all out.

this fileset would be a binary packed format, like CrAU?

@vbatts

@vbatts https://github.com/stevvooe/continuity has been opened up to continue this research.

Probably related to #302. Will need to be considered as part of that.

I think the https://github.com/GNOME/ostree format has a lot of advantages. It was designed from the start to be checksummed. If implementing anything else, at least study it.

@philips

On Thu, Feb 25, 2016 at 01:30:26PM -0800, Colin Walters wrote:

Going back to the top of the motivation here:

Problem: a rootfs for a container bundle sitting on-disk may not
reflect the exact intended state of the bundle when it was copied
to its current location.

I'd say the correct solution here is for the container runtime to
work with the storage layer to ensure immutability.

The O_OBJECT proposal you link is about preserving filesystem content
after it lands on the filesystem, but it looks like @philips initial
concern was about landing it on the filesystem in the first place.
For example, “my FAT-16 filesystem doesn't support POSIX permissions,
so my rootfs/foo/bar seems to have 0777 instead of the source's 0600”.

I struggle to understand a scenario where one would reasonably want to unpack container content onto FAT-16 and expect to run it. Maybe inspection, but even then, you can do that from userspace easily enough with libarchive or whatever. If you have a Linux container, you have Linux...hence you have xfs/ext4/etc.

@philips

On Thu, Feb 25, 2016 at 03:28:36PM -0800, Colin Walters wrote:

I struggle to understand a scenario where one would reasonably want
to unpack container content onto FAT-16…

A poor choice of example, but @philips was pointing out that not all
filesystems support the same attributes (he pointed out NFS without
xattrs, among other things). Regardless of the specific examples,
unpacking into a local filesystem (what @philips was talking about
1) and maintaining content after that unpacking (what you were
talking about 2) are two separate things.

Anyways my goal here is to try to ensure sharing of ideas, not necessarily code in this area - OSTree is certainly not going to take over the world as a way to get content from A to B any more than other projects in this area. Another good project to look at is Clear Linux: https://lists.clearlinux.org/pipermail/dev/2016-January/000159.html

A good example of a mistake in OSTree - I've come to realize the git-like Merkle tree model was a mistake for binary content, because it's really common with software updates for one "package" to change multiple paths (due to /usr/bin and /usr/share etc.) For git and source code it's a lot more common to only change one subdirectory.

So the Clear Linux manifest makes sense - there's no streaming, but that's fine because we aren't storing huge amounts of content to tape drives.

Also, OSTree not including the size in the tree metadata was really dumb but that's papered over with static deltas.

Speaking of deltas...that's another area where Docker really lacks, and for OSTree I ended up taking a ton of inspiration from http://dev.chromium.org/chromium-os/chromiumos-design-docs/filesystem-autoupdate. For more on that see https://ostree.readthedocs.org/en/latest/manual/formats/

Regarding NFS...sure, but how does it help a user/admin to determine after the fact that things are broken? Basically the system is either going to munch the fscaps on /bin/ping or not, a system that tells you "hey the fscaps are missing" may lead you Google faster but that's about it...

fscap binaries can be pretty easily worked around in an NFS root scenario by copying them into tmpfs or something on boot. Yes, it's ugly, see: https://bugzilla.redhat.com/show_bug.cgi?id=648654#c19

Also, I'd like to go on a crusade to kill off setuid binaries in containers - they're legacy, and in a container world we should always run with NO_NEW_PRIVS on. Use containers as a reason to leave behind the continual security issues of setuid, and just have them on the host until someone rewrites PAM and /sbin/unix_chkpwd etc.

@philips

On Thu, Feb 25, 2016 at 06:31:09PM -0800, Colin Walters wrote:

Regarding NFS...sure, but how does it help a user/admin to determine
after the fact that things are broken? Basically the system is
either going to munch the fscaps on /bin/ping or not, a system
that tells you "hey the fscaps are missing" may lead you Google
faster but that's about it...

Agreed if the goal is going image → filesystem → running container,
but I think @philips was concerned with round-tripping from image
files to filesystem bundles (image 1 → filesystem → image 2), since he
links tar-split which is focused on unpacking and repacking tarballs
while preserving the tarball's hash. Folks that are interested in
round-tripping through the filesystem would be concerned about
mismatches between attributes represented in the filesystem and
attributes represented in the image file, but not about freezing
content once it's on the filesystem. And folks that want to
round-trip in the face of limited filesystems can write tools that
stash the unsupported attributes elsewhere and pull them back in when
checking for changes, so they can do better than failing fast.

Personally, I don't think round-tripping is particularly useful,
because:

Folks who just want to verify a bundle can fetch the original image
[1](e.g. by caching it locally) so they don't have to regenerate
the original image from the filesystem.
Folks who want to generate a new image that reuses content
addressable objects from an earlier image (e.g. adding a few files
to a stock Debian image to create a new image) can handle that
locally (e.g. with something like Git's staging area to bless
changes they're interested in). There's no need to address this at
the protocol / file-format level.

Subject: Re: OCI Bundle Digests Summary
Date: Thu, 15 Oct 2015 16:52:42 -0700
Message-ID: 20151015235242.GD28418@odin.tremily.us

I am closing this out. The image format work is now part of the OCI Image Format project: https://github.com/opencontainers/image-spec

I have an aside question: if there is a bug Undefined syscall.TIOCGPTN, syscall.TIOCSPTLCK during compilation of cri-o (gollvm related)
, related to https://github.com/containerd/containerd/tree/master/vendor/github.com/containerd/continuity/fs - where should I open issues/report ?

Ivan

@advancedwebdeveloper

@advancedwebdeveloper source code of that package is in https://github.com/containerd/continuity

Navigation Menu

Repository navigation

bundle: filesystem metadata format #11

Basic Themes

Statements in general I've found true in crypto engineering. May be useful in this context.

- Define our threat model before we do anything else. Who are we trying to protect against?

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bundle: filesystem metadata format #11

Description

Activity

stevvooe commented on Jun 29, 2015

philips commented on Jun 30, 2015

stevvooe commented on Jun 30, 2015

philips commented on Jun 30, 2015

stevvooe commented on Jun 30, 2015

philips commented on Jul 1, 2015

stevvooe commented on Jul 1, 2015

stevvooe commented on Jul 1, 2015

bitshark commented on Jul 3, 2015

Basic Themes

Statements in general I've found true in crypto engineering. May be useful in this context.

- Define our threat model before we do anything else. Who are we trying to protect against?

vbatts commented on Aug 13, 2015

stevvooe commented on Sep 2, 2015

duglin commented on Jan 13, 2016

cgwalters commented on Feb 12, 2016

7 remaining items

wking commented on Feb 25, 2016

cgwalters commented on Feb 25, 2016

wking commented on Feb 25, 2016

cgwalters commented on Feb 26, 2016

cgwalters commented on Feb 26, 2016

cgwalters commented on Feb 26, 2016

wking commented on Feb 26, 2016

philips commented on Apr 6, 2016

advancedwebdeveloper commented on Sep 16, 2020

thaJeztah commented on Sep 16, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions