Skip to content

Migrate the scripts in S3 to use script._id instead of the path value #486

Open
Open
Migrate the scripts in S3 to use script._id instead of the path value#486
Labels
CODESome other Code related issue and it should clearly describe what it is affecting in a comment.DBPertains inclusively to the Database operations.bugYou've guessed it... this means a bug is reported.migrationUse this to indicate that it may apply to an existing or announced migration.needs discussionBlah, blah, blah, wahh, wahh, wahh, etc.
Milestone
#486

Description

@Martii
Martii
opened on Dec 6, 2014

Currently we are unable to upgrade the aws-sdk dependency because "something" changed that makes getObject to fail.

This has been explained in detail at aws/aws-sdk-js#430 . With all the test deploys done trying to figure out what the steps to reproduce are it appears that normalizing the casing when we save to the DB is an option.

Looking for more options related to advancing the goal of updating the dep... e.g. perhaps we could validate the current aReq.params.username against the real username (name) that is typically in the Raw JSON Data for an account. e.g. a lookup to correct the casing around here... but I suspect this is going to need to be a DB migration with a BLOCKING label until it is finished.

See also:

Test points:

Activity

  1. Martii
    added
    DBPertains inclusively to the Database operations.
    bugYou've guessed it... this means a bug is reported.
    CODESome other Code related issue and it should clearly describe what it is affecting in a comment.
    needs discussionBlah, blah, blah, wahh, wahh, wahh, etc.
    migrationUse this to indicate that it may apply to an existing or announced migration.
    team bizThis is similar to a meta discussion.
    on Dec 6, 2014
  2. Martii
    added a commit that references this issue on Dec 6, 2014

    Graceful failure when S3 *(fake or real)* doesn't have key

    e2bca6d

  3. Martii commented on Dec 6, 2014

    @Martii
    Martii
    on Dec 6, 2014
    MemberAuthor

    With #487 we have another option of inline live migration... that patch currently fixes the net timeout in dev when selecting some other script source with fakeS3 and it usually just spins the browser spinner for an incredibly long time since our local dev DB doesn't usually have that source.

    We can either correct the casing to what the user entered, if applicable, (e.g. a quick user search in the DB and if it is a hit compare it to the username portion generated from installName and migrate) from what we store in the Raw JSON Data or we can just lowercase migrate everything username wise... It is probably not recommended that we should alter the scriptname value though in casing with the Key.

    Updated the above for some test points and other linkage.

  4. Zren commented on Dec 6, 2014

    @Zren
    Zren
    on Dec 6, 2014
    Contributor

    Normally you wouldn't base the filename/keys based on the username/scriptname. Create an id for each source code and store it in the Script model. If we wanted to future proof ourselves, we wouldn't use the Script._id either as that means only 1 source code per Script. But since we're not versioning it atm, we could just use that.

    We have to fetch the Script model first before we doing any AWS stuff anyways.

    Edit: Also noticed this when I tried to refactor the scriptEditSourceCodePage that never got finished. Most of the callbacks need to follow the callback(err, ...) pattern so we can chain error up to the route handlers.

  5. sizzlemctwizzle commented on Dec 6, 2014

    @sizzlemctwizzle
    sizzlemctwizzle
    on Dec 6, 2014
    Member

    I agree with @Zren. We should use Script._id instead since we have to do a lookup anyway.

  6. Martii commented on Dec 7, 2014

    @Martii
    Martii
    on Dec 7, 2014
    MemberAuthor

    Create an id for each source code and store it in the Script model.

    Yah I'm not entirely sure how lower casing a Unicode character would affect things so a unique id would probably be best. EDIT: Searching will need to be retested for this as well.

  7. Martii commented on Dec 9, 2014

    @Martii
    Martii
    on Dec 9, 2014
    MemberAuthor

    Just a FYI here's a short snippet from recent production log with the isPro console message I added in to check for this on pro e.g. the version of aws-sdk we are currently using hides/mishandles the error and possibly returns an empty payload as one of the maintainers stated... jri's script does show source and does install.

    [12/08 23:36:21 EST][err] S3 Key Not Found jri/Geocaching_Map_Enhancements.user.js
[12/08 23:40:33 EST][out] Group(imgur) Rating Updated
[12/08 23:45:24 EST][out] Group(deviantART) Rating Updated
[12/08 23:48:10 EST][err] S3 Key Not Found mienaikage/OkChoices.user.js
[12/08 23:48:53 EST][out] Group(Subtitles) Rating Updated
[12/08 23:49:42 EST][out] Group(Steam) Rating Updated
[12/08 23:49:42 EST][out] Group(dota2lounge) Rating Updated
[12/08 23:49:47 EST][out] Group(download) Rating Updated
[12/08 23:51:02 EST][out] GitHub client authenticated
[12/08 23:51:10 EST][out] Group(bookmarks) Rating Updated

    Hmmm their clock appears to be a bit skewed too.

  8. Martii commented on Apr 27, 2015

    @Martii
    Martii
    on Apr 27, 2015
    MemberAuthor

    Hmmm... retested this randomly with aws-sdk@2.1.25 and doesn't appear to crash dev nor local pro anymore... not sure on actual production. All prior test points appear to be okay (without crashing local pro). :\

    @sizzlemctwizzle
    Any objections to trying it on production? I'll give this a day or so for any responses.

  9. Martii
    added a commit that references this issue on Apr 27, 2015

    Try for *aws-sdk-js* update

    82342eb

  10. sizzlemctwizzle commented on Apr 27, 2015

    @sizzlemctwizzle
    sizzlemctwizzle
    on Apr 27, 2015
    Member

    I have no problem with a temporary fix, but we definitely need to migrate the scripts in S3 to use script._id instead of the path value somewhere down the line. It's a simple migration but I'm always afraid I'm going to break something when I mess with S3. I spent hours trying to get everything right when I removed namespaces.

  11. 5 remaining items

  12. Martii
    removed
    needs discussionBlah, blah, blah, wahh, wahh, wahh, etc.
    on Jun 29, 2015
  13. Martii
    added this to the #486 milestone on Jul 2, 2015
  14. Martii
    added 2 commits that reference this issue on Jul 7, 2015

    Hot fix for OpenUserJS#656

    372c885

    Missed STYLEGUIDE.md conformance

    02fccb1
  15. Martii
    added a commit that references this issue on Aug 18, 2015

    Some misc STYLEGUIDE.md conformance in `.../scriptStorage.js`

    d433f0a
  16. Martii
    added a commit that references this issue on Dec 16, 2015

    Change some defaults with MongoDB and stderr

    bb16868
  17. Martii
    added
    needs discussionBlah, blah, blah, wahh, wahh, wahh, etc.
    and removed
    soonerSooner would be appreciated.
    on Apr 19, 2017

  18. Martii commented on Apr 19, 2017

    @Martii
    Martii
    on Apr 19, 2017
    MemberAuthor

    There's a major consequence to doing this... if script._id is ever used as the key in S3, then if the DB ever needs to be "moved" somewhere all of the _ids could be different depending on how it was restored. If it's a DB backup and then a restore then it might be okay if it preserves the same _ids; but if it's a copy/delete op all of the _ids will be different and all source will be orphaned.

    NOTE: This is happening less and less as time progresses and fixes roll through.

  19. Martii
    added 4 commits that reference this issue on Apr 20, 2017

    Ensure casing on username is identical for S3 retrieval from DB

    4d744d6

    Ensure casing on username is identical for S3 retrieval from DB (#1086)

    583c4a5

    Clean up the console message for possible S3 errors

    00582c5

    Clean up the console message for possible S3 errors (#1094)

    ddab15a
  20. Martii
    added 2 commits that reference this issue on May 1, 2017

    Invert remote saves

    9f4a8ca

    Invert remote saves (#1098)

    1424dce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CODESome other Code related issue and it should clearly describe what it is affecting in a comment.DBPertains inclusively to the Database operations.bugYou've guessed it... this means a bug is reported.migrationUse this to indicate that it may apply to an existing or announced migration.needs discussionBlah, blah, blah, wahh, wahh, wahh, etc.

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions