You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Bug 27755 - Using the Subtle Crypto Interface with Streams #73
Though the StreamsAPI is referenced in Informative Reference, the functions under window.crypto.subtle are specified with only one-shot data inputs.
Use-cases: Data may not be available at once. Data may be too huge to keep in memory.
For encrypt()/decrypt() it would make sense to have a streaming readable output if the input is a readable stream.
Reacted by Feross Aboukhadijeh, Mitar, Fedor Indutny, Sindre Sorhus, Tankred Hase, George Miroshnykov, Justin, Giovanni Pellerano, Ram, Gustavo Rodrigues and 123 more
After listening to Ryan rage about the use of BER encoding for ASN.1 objects, I have a feeling that this should be closed as won't fix because it presents a security issue. When one looks at the encrypt/decrypt APIs for authenticated encryption, it is required that the entire stream be observed on the decrypt side and could be argued that it needs to be observed on the encrypt side prior to emitting the processed stream. This is due to the fact that if the decryption process does not validate then no output is to be produced for consumption. Allowing this to be done in a streaming fashion means that the browser potentially needs to have an infinite size buffer to hold the intermediate result to be returned to the client.
Similar issues hold for processing of signature values for the new X448 EdDSA algorithm where the message M is hashed twice. Allowing for an indefinite length input means that there are potential buffer overrun problems.
Node.js has a streaming crypto API without any security issues:
constcrypto=require('crypto');consthash=crypto.createHash('sha256');hash.update('some data to hash');hash.update('more data');hash.update('even more data');console.log(hash.digest('hex'));
Why can't the web platform?
Reacted by Jason Kurian, Luka Vidaković, Gustavo Rodrigues, Rocco Musolino, Nathan Ginnever, Etienne Martin, Karthik Kailash, David Dias, John Ozbay, Wiktor Kwapisiewicz and 61 more
Reacted by David Dias, William Blankenship, Nick, Joel Hernández, Matthew Little, Gus Narea, Ben Brook, Stefan Otto, Dr Ian Preston, Matthew and 5 more
I absolutely agree with @feross on this. Most (if not all) of the APIs can work in a streaming mode without any security issues. In fact, this is how these APIs are exposed in OpenSSL, so they always work in a streaming mode under the hood anyway, regardless of what high-level API may look like.
Reacted by Karthik Kailash, Matthew Little, Ben Brook, Pardal, Matthew, Brad Isbell, nitn3lav, Greg Slepak, devgs, Martin Heidegger and 2 more
Reacted by David Dias, Matthew Little, Matthew and martinml
All of the current hash functions that I am familiar with will allow for streaming APIs because they are built using a Merkle–Damgård construction. This means that they are processed on a block by block basis. However there are algorithms for which this is not doable. For example, the EdDSA algorithm that I mentioned above computes:
R = fn( SHAKE256(dom(F, C) || prefix || M, 114) )
and then
k = SHAKE256(dom(F, C) || R || A || M, 114)
as you can see, you need to all of the message M to compute R before you can start doing the computation of k. This means that the entire message needs to be buffered unlike the hash example you gave above.
Note also the comment that I made on authenticated decryption where the entire message needs to be kept before doing the validation step at the end.
Reacted by sideshowbarker, Dave Longley, nitn3lav, aggregate1166877 and olfek
@jimsch in your description SHAKE256 appears to be just a hashing function, most of the hashing functions support streaming input. There is nothing that could prevent one from creating two streaming SHAKE256 hashes and using their digests at the end of the stream to compute R and k.
Authenticated decryption should work as well, as far as I can tell... Though, the fact that the integrity is checked only at the end of decryption process means that the API will be kind of awkward. I don't think that there are much pros of using streams for authenticated decryption.
I imagine we can close this as won't fix, but when streaming stabilizes we can then revisit as part of maintenance of the spec since as @jimsch correctly points out, it won't work for quite a few algorithms. We could also try to test to see if anyone supports streaming - any ideas?
Reacted by KittenWithHerbs, Jimmy Wärting, Steven Goris, Philipp Bauer, Matthew Little, Tosho Toshev, Ben Brook, T6, nitn3lav, devgs and 4 more
If streaming/progressive encryption isn't implemented, it's going to hugely limit the scope of usage of the API. I really need that kind of functionality for the software I work on.
Reacted by Etienne Martin, Nathan Ginnever, Karthik Kailash, John Ozbay, Tyler Biscoe, KittenWithHerbs, Casey Rojas, Chris Adams, Philipp Bauer, Ben Brook and 10 more
Well - with no offense to the OpenSSL team, I don't think we should take their API as the end-all and be-all of crypto APIs.
AEAD modes are typically defined as single-shot functions, that don't allow passing a stream. Deviating from that would need a stronger justification than "OpenSSL does it", IMHO.
Indeed many libraries do not expose streaming AEADs, including some of those used by browsers. OpenSSL's streaming API is an artifact of them trying to mix multiple, unrelated algorithms into the same API. (A similar mistake that has been made here.)
Any proposal with a streaming AEAD is a non-starter here. AEADs are simply not streaming primitives.
Also, due to the current state of cryptography algorithms and their usage, stream input to hashes is a bit more critical.
Most symmetric encryption currently uses AES, a block cipher. The streaming modes for block ciphers like AES could mostly be done in userland as (aside from GCM and GCM-SIV) most of them are very straightforward to implement atop sync algorithms. GCM is trickier and requires a bit of math, and counter mode requires an awkward carry calculation, but the rest of the common ones are just key-width XORs at specific times.
Asymmetric encryption usually operates on whole buffers in practice, as the relevant messages are normally only kilobytes long.
Hashing is very frequently done over streams of indefinite length. S3's API for instance accepts an x-amz-checksum-{ALGO} header where two of the algorithms available are SHA-1 and SHA-256. This, for a chunk of several gigabytes, would necessitate a streaming hash before uploading in most browsers.
Reacted by abnessor aka findoff, Brad Isbell and Benjamin Goering
Unfortunately, the authors and core proponents of this spec couldn't care less about hashing use cases outside of crypto. We've hashed over this (heh) on previous threads while trying to get something as simple as SHA-1 on "insecure" origins:
Therefore, I doubt we'll get built-in hashing of streams without the associated crypto side... but who knows, maybe someone will prove me wrong. We're stuck with DIY in the meantime. At least it's possible.
@bradisbell Bringing in unrelated issues isn't really relevant, and also not really helpful while we're actively discussing the addition of streaming to Web Crypto.
Taking a step back, and perhaps contrary to what I said before in this thread; if we want to focus on adding streaming hashing first since it's least controversial, and if everyone is happy with the proposal in #73 (comment) (i.e. extending the existing function rather than adding a new DigestStream object or some such), perhaps we don't strictly need to have a separate draft in WICG for that, and we could just directly make a PR here?
If folks are OK with that, I can write something up.
Reacted by Claudia Meadows, Brad Isbell, James M Snell, Brad Marsden and Brian M Hunt
... and if everyone is happy with the proposal in #73 (comment) (i.e. extending the existing function rather than adding a new DigestStream object or some such),
@jakearchibald's proposal looks good to me. Workers will need to keep DigestStream around forever as we don't make breaking changes but having the existing crypto.subtle.digest accept a ReadableStream as input would work just as well for our uses cases. Actually, I would go one step further and say that instead of just accepting a ReadableStream, allow it to accept any Iterable and AsyncIterable.
I imagine that crypto.subtle.sign and crypto.subtle.verify could both also follow the same pattern without much controversy.
Reacted by Daniel Huigens, Claudia Meadows, Benjamin Goering, Brian M Hunt, thrasher and Martin Heidegger
Just adding a real world use case: WhatsApp API webhook's requests are signed with the hash of the request body and a secret, so in order to verify the request with the current API, the entire body need to be copied to another array to concatenate the secret, a waste of memory resources.
And now an example of an alternative API, Cloudfare Workers adds the non standard DigestStream in order to solve this kind of problems.
Another use-case is user-encrypted file storage. For example, uploading a 100GB file encrypted with your password client-side would require streaming encryption, and conversely streaming decryption to download it.
Encryption is unfortunately lacking a standard for this. It's probably a bad idea to just encrypt the whole 100gib as it makes range queries impossible. A more flexible approach is to break it down into smaller chunks (eg 64kib at a time) and encrypt each of those before sending. That kind of model of course doesn't require any of this work although a standardized interface and implementation would remove a lot of the complexity from doing it and taking advantage of it.
Reacted by Claudia Meadows, Zefir, thrasher and Chris Cowan
@bencmbrook That use case is exactly where the streaming proposal here would not work. That is not to say there isn't a use case for streaming encryption. Obviously we have streaming constructions for encrypting things everywhere, including TLS. However, the secure way to build them is not to break apart the individual encryption primitives, but to use those primitives to encrypt chunks. This is because those primitives are designed to encrypt and authenticate individual atomic messages.
(Doing that correctly also takes some care. The sequence number needs to be fed into the encryption, often as the AEAD's IV/nonce. Having that in the platform wouldn't be wrong, but would need folks to agree on a standard construction, and alas protocols tend to build this component themselves.)
It looks like there's been some positive progress on this over the last few months which is awesome! That said, it's hard to tell from disparate PRs what the up-to-date summary is... Would any of you happen to know the latest summary? Thanks in advance. :-)
We're stuck with DIY in the meantime. At least it's possible.
Unfortunately, DIY is at best 3x slower than native. That's even with the most absolutely hand-optimized wasm builds of various hashing algorithms. (See Daninet/hash-wasm)
Activity
jimsch commentedon May 24, 2016
After listening to Ryan rage about the use of BER encoding for ASN.1 objects, I have a feeling that this should be closed as won't fix because it presents a security issue. When one looks at the encrypt/decrypt APIs for authenticated encryption, it is required that the entire stream be observed on the decrypt side and could be argued that it needs to be observed on the encrypt side prior to emitting the processed stream. This is due to the fact that if the decryption process does not validate then no output is to be produced for consumption. Allowing this to be done in a streaming fashion means that the browser potentially needs to have an infinite size buffer to hold the intermediate result to be returned to the client.
Similar issues hold for processing of signature values for the new X448 EdDSA algorithm where the message M is hashed twice. Allowing for an indefinite length input means that there are potential buffer overrun problems.
feross commentedon May 24, 2016
Node.js has a streaming crypto API without any security issues:
Why can't the web platform?
indutny commentedon May 24, 2016
I absolutely agree with @feross on this. Most (if not all) of the APIs can work in a streaming mode without any security issues. In fact, this is how these APIs are exposed in OpenSSL, so they always work in a streaming mode under the hood anyway, regardless of what high-level API may look like.
jimsch commentedon May 24, 2016
All of the current hash functions that I am familiar with will allow for streaming APIs because they are built using a Merkle–Damgård construction. This means that they are processed on a block by block basis. However there are algorithms for which this is not doable. For example, the EdDSA algorithm that I mentioned above computes:
R = fn( SHAKE256(dom(F, C) || prefix || M, 114) )
and then
k = SHAKE256(dom(F, C) || R || A || M, 114)
as you can see, you need to all of the message M to compute R before you can start doing the computation of k. This means that the entire message needs to be buffered unlike the hash example you gave above.
Note also the comment that I made on authenticated decryption where the entire message needs to be kept before doing the validation step at the end.
indutny commentedon May 24, 2016
@jimsch in your description
SHAKE256appears to be just a hashing function, most of the hashing functions support streaming input. There is nothing that could prevent one from creating two streamingSHAKE256hashes and using their digests at the end of the stream to computeRandk.Authenticated decryption should work as well, as far as I can tell... Though, the fact that the integrity is checked only at the end of decryption process means that the API will be kind of awkward. I don't think that there are much pros of using streams for authenticated decryption.
jimsch commentedon May 24, 2016
@indutny please re-read my previous post and look at the requirements to finish R before using M for k
indutny commentedon May 24, 2016
@jimsch oh I see it now. Sorry about that! Yeah, streaming won't work for this kind of encryption/decryption schemes indeed.
Still many hashes and ciphers work just fine with streams.
tanx commentedon May 24, 2016
A native streaming api would indeed be great. Our use-case would be large file encryption in OpenPGP.js.
mwatson2 commentedon May 24, 2016
If we address this, I think it will not be in this version since it requires substantial work.
hhalpin commentedon Jun 20, 2016
I imagine we can close this as won't fix, but when streaming stabilizes we can then revisit as part of maintenance of the spec since as @jimsch correctly points out, it won't work for quite a few algorithms. We could also try to test to see if anyone supports streaming - any ideas?
hhalpin commentedon Jun 20, 2016
v.Next.
evilaliv3 commentedon Dec 13, 2016
Is there any update on this topic?
roccomuso commentedon Mar 6, 2017
+1
ericmackrodt commentedon Mar 30, 2017
If streaming/progressive encryption isn't implemented, it's going to hugely limit the scope of usage of the API. I really need that kind of functionality for the software I work on.
77 remaining items
twiss commentedon Nov 21, 2024
Well - with no offense to the OpenSSL team, I don't think we should take their API as the end-all and be-all of crypto APIs.
AEAD modes are typically defined as single-shot functions, that don't allow passing a stream. Deviating from that would need a stronger justification than "OpenSSL does it", IMHO.
davidben commentedon Nov 21, 2024
Indeed many libraries do not expose streaming AEADs, including some of those used by browsers. OpenSSL's streaming API is an artifact of them trying to mix multiple, unrelated algorithms into the same API. (A similar mistake that has been made here.)
Any proposal with a streaming AEAD is a non-starter here. AEADs are simply not streaming primitives.
knightcode commentedon Nov 21, 2024
I don't mean to stir trouble, but you didn't address my point. It was not "OpenSSL does it." It was "it's not your decision to make."
dead-claudia commentedon Nov 21, 2024
Also, due to the current state of cryptography algorithms and their usage, stream input to hashes is a bit more critical.
x-amz-checksum-{ALGO}header where two of the algorithms available are SHA-1 and SHA-256. This, for a chunk of several gigabytes, would necessitate a streaming hash before uploading in most browsers.bradisbell commentedon Nov 22, 2024
Unfortunately, the authors and core proponents of this spec couldn't care less about hashing use cases outside of crypto. We've hashed over this (heh) on previous threads while trying to get something as simple as SHA-1 on "insecure" origins:
The key comment from @twiss:
#170 (comment)
Therefore, I doubt we'll get built-in hashing of streams without the associated crypto side... but who knows, maybe someone will prove me wrong. We're stuck with DIY in the meantime. At least it's possible.
twiss commentedon Nov 22, 2024
@bradisbell Bringing in unrelated issues isn't really relevant, and also not really helpful while we're actively discussing the addition of streaming to Web Crypto.
@dead-claudia and @jpsugar-flow I tend to agree.
Taking a step back, and perhaps contrary to what I said before in this thread; if we want to focus on adding streaming hashing first since it's least controversial, and if everyone is happy with the proposal in #73 (comment) (i.e. extending the existing function rather than adding a new
DigestStreamobject or some such), perhaps we don't strictly need to have a separate draft in WICG for that, and we could just directly make a PR here?If folks are OK with that, I can write something up.
jasnell commentedon Nov 22, 2024
@jakearchibald's proposal looks good to me. Workers will need to keep
DigestStreamaround forever as we don't make breaking changes but having the existingcrypto.subtle.digestaccept aReadableStreamas input would work just as well for our uses cases. Actually, I would go one step further and say that instead of just accepting aReadableStream, allow it to accept anyIterableandAsyncIterable.I imagine that
crypto.subtle.signandcrypto.subtle.verifycould both also follow the same pattern without much controversy.taralx commentedon Nov 22, 2024
Oh, I missed that proposal. Very elegant. +1 from me. (I'm jpsugar-flow, but this is not work-related for me.)
robmv commentedon Jan 19, 2025
Just adding a real world use case: WhatsApp API webhook's requests are signed with the hash of the request body and a secret, so in order to verify the request with the current API, the entire body need to be copied to another array to concatenate the secret, a waste of memory resources.
And now an example of an alternative API, Cloudfare Workers adds the non standard DigestStream in order to solve this kind of problems.
bencmbrook commentedon Jan 21, 2025
Another use-case is user-encrypted file storage. For example, uploading a 100GB file encrypted with your password client-side would require streaming encryption, and conversely streaming decryption to download it.
vlovich commentedon Jan 21, 2025
Encryption is unfortunately lacking a standard for this. It's probably a bad idea to just encrypt the whole 100gib as it makes range queries impossible. A more flexible approach is to break it down into smaller chunks (eg 64kib at a time) and encrypt each of those before sending. That kind of model of course doesn't require any of this work although a standardized interface and implementation would remove a lot of the complexity from doing it and taking advantage of it.
davidben commentedon Jan 21, 2025
@bencmbrook That use case is exactly where the streaming proposal here would not work. That is not to say there isn't a use case for streaming encryption. Obviously we have streaming constructions for encrypting things everywhere, including TLS. However, the secure way to build them is not to break apart the individual encryption primitives, but to use those primitives to encrypt chunks. This is because those primitives are designed to encrypt and authenticate individual atomic messages.
(Doing that correctly also takes some care. The sequence number needs to be fed into the encryption, often as the AEAD's IV/nonce. Having that in the platform wouldn't be wrong, but would need folks to agree on a standard construction, and alas protocols tend to build this component themselves.)
See these links for some background:
https://www.imperialviolet.org/2014/06/27/streamingencryption.html
https://www.imperialviolet.org/2015/05/16/aeads.html
pthrasher commentedon May 25, 2025
@davidben, @twiss, or @jasnell
It looks like there's been some positive progress on this over the last few months which is awesome! That said, it's hard to tell from disparate PRs what the up-to-date summary is... Would any of you happen to know the latest summary? Thanks in advance. :-)
Unfortunately, DIY is at best 3x slower than native. That's even with the most absolutely hand-optimized wasm builds of various hashing algorithms. (See Daninet/hash-wasm)
twiss commentedon May 28, 2025
@pthrasher #390 needs to be adapted to whatwg/webidl#1397, which in turn may need to be changed: whatwg/webidl#1490. I'll try to get to it once WebIDL is stable :)