snapshot: trim extension statements from pgdump when uploading - #893
Conversation
c609124 to
15b2d10
Compare
| line, err = reader.ReadBytes('\n') | ||
| consumed += int64(len(line)) | ||
| if err != nil { | ||
| return |
There was a problem hiding this comment.
Style nit: I was surprised to not see an err here or in the rest of the function
There was a problem hiding this comment.
+1 nit: not a fan of naked return... 😛
| // Once we start seeing table creations, we are definitely done with extensions, | ||
| // so we can hand off the rest to the superior io.Copy implementation. |
There was a problem hiding this comment.
Are we sure about this? I'm lacking some context if this is being used by end users or not but this is a large assumption.
The official postgres docs use things like split https://www.postgresql.org/docs/12/backup-dump.html#BACKUP-DUMP-LARGE
There was a problem hiding this comment.
If you're referring to table creations being before extensions, then yes - extensions are typically a prerequisite
As for handing off to io.Copy, yes - the io.Copy implementation is more robust and efficient, and the GCS bucket handler will handle chunking for us. I don't think Cloud SQL will be happy with split dumps, and to piece it together we need to download it somewhere, which I'd like to avoid (right now the process can just have us import directly from GCS)
There was a problem hiding this comment.
I think we can revisit things like split if size becomes a blocker, but I think the streaming nature of io.Copy should mitigate the effects of size
When uploading a snapshot, trim extension statements unsupported by Google Cloud SQL. We do this on upload so that the customer's copy of the dump remains unmodified (in case something goes wrong, we still have the original to work with - trimming can be disabled with trim-extensions=false).
3 participants
When uploading a snapshot, trim extension statements unsupported by Google Cloud SQL. We do this on upload so that the customer's copy of the dump remains unmodified (in case something goes wrong, we still have the original to work with - trimming can be disabled with
trim-extensions=false).Closes https://github.com/sourcegraph/customer/issues/1613
Test plan
See uploaded data