Skip to content

users prune - Paginate graphql site users - fix bug for soft deleted users - #909

Merged
DaedalusG merged 10 commits into
mainfrom
paginate-users-prune
Jan 16, 2023
Merged

users prune - Paginate graphql site users - fix bug for soft deleted users#909
DaedalusG merged 10 commits into
mainfrom
paginate-users-prune

Conversation

@DaedalusG

@DaedalusG DaedalusG commented Jan 4, 2023

Copy link
Copy Markdown
Contributor

This PR fixes an issue where user removal graphQL queries were issued to delete users who had already been soft deleted (these users are returned by the site.users graphql api)

It also adds pagination to pull 100 users per graphql request until all users in site.users have been checked against removal conditions

For a better UX on large instances printing a table of users to be deleted during the confirmation check to delete users has been placed behind a flag

This PR is primarily a bug fix since currently the command is broken -- further documentation of the command will be added when its more robust (for instance allowing input of a json user exclusion list, and the addition of unit tests)

λ ~/src-cli/ paginate-users-prune* src users prune -h
Usage of 'src users prune':
  -days int
    	Days threshold on which to remove users, must be 60 days or greater and defaults to this value  (default 60)
  -display-users
    	display table of users to be deleted by prune
  -dump-requests
    	Log GraphQL requests and responses to stdout
  -force
    	skips user confirmation step allowing programmatic use
  -get-curl
    	Print the curl command for executing this query and exit (WARNING: includes printing your access token!)
  -insecure-skip-verify
    	Skip validation of TLS certificates against trusted chains
  -remove-admin
    	prune admin accounts
  -remove-null-users
    	removes users with no last active value
  -trace
    	Log the trace ID for requests. See https://docs.sourcegraph.com/admin/observability/tracing
  -user-agent-telemetry
    	Include the operating system and architecture in the User-Agent sent with requests to Sourcegraph (default true)

This command removes users from a Sourcegraph instance who have been inactive for 60 or more days. Admin accounts are omitted by default.

Examples:

	$ src users prune -days 182

	$ src users prune -remove-admin -remove-null-users

λ ~/src-cli/ paginate-users-prune* src users prune
49 of 209 users were inactive for more than 60 days on https://k8s.sgdev.org.
Do you  wish to proceed with user removal [y/N]: n
Aborting removal
λ ~/src-cli/ paginate-users-prune* src users prune -display-users
Users to remove from https://k8s.sgdev.org
╭─────────────────────────┬─────────────────────────────────────────┬────────────────────────╮
│ USERNAME                │ EMAIL                                   │ DAYS SINCE LAST ACTIVE │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ rijnard                 │ rijnard@sourcegraph.com                 │                    202 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ christine               │ christine@sourcegraph.com               │                    211 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ christina               │ christina@sourcegraph.com               │                    124 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ mike                    │ mike@sourcegraph.com                    │                     91 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jean                    │ jean@sourcegraph.com                    │                    206 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ loic                    │ loic@sourcegraph.com                    │                    117 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jonah                   │ jonah@sourcegraph.com                   │                    154 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ mariam                  │ mariam@sourcegraph.com                  │                    120 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ emily                   │ emily@sourcegraph.com                   │                    198 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ maria                   │ maria@sourcegraph.com                   │                    232 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ afogg                   │ afogg@sourcegraph.com                   │                    211 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ ben.venker              │ ben.venker@sourcegraph.com              │                    145 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ felix.kling             │ felix.kling@sourcegraph.com             │                     89 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ beatrix-test            │ buildkite@perforce.sgdev.org            │                     82 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ mohammad.alam           │ mohammad.alam@sourcegraph.com           │                    173 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ elzanne.wentzel         │ elzanne.wentzel@sourcegraph.com         │                    170 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ chrispine               │ chrispine@sourcegraph.com               │                    196 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nick                    │ nick@sourcegraph.com                    │                    148 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ filip.haftek            │ filip.haftek@sourcegraph.com            │                    146 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ alex.jean-baptiste      │ alex.jean-baptiste@sourcegraph.com      │                     84 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ michal.vrtiak           │ michal.vrtiak@sourcegraph.com           │                     91 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ taylor.sperry           │ taylor.sperry@sourcegraph.com           │                     75 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ justin.dorfman          │ justin.dorfman@sourcegraph.com          │                    100 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ ajay.sridhar            │ ajay.sridhar@sourcegraph.com            │                    238 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nate.freng              │ nate.freng@sourcegraph.com              │                    230 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ tech-ops-admin          │ tech-ops-admin@sourcegraph.com          │                     79 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ michelle.veronese       │ michelle.veronese@sourcegraph.com       │                     68 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jeff.wayman             │ jeff.wayman@sourcegraph.com             │                    218 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ marisa.kanemoto         │ marisa.kanemoto@sourcegraph.com         │                    208 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ rakesh.joshi            │ rakesh.joshi@sourcegraph.com            │                    218 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ kristen.stretch         │ kristen.stretch@sourcegraph.com         │                    239 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ qkeast                  │ qkeast@sourcegraph.com                  │                    105 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ joe.johnson             │ joe.johnson@sourcegraph.com             │                    236 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ james.cotter            │ james.cotter@sourcegraph.com            │                    189 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ isuru.fonseka           │ isuru.fonseka@sourcegraph.com           │                    231 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ aditya.kalia            │ aditya.kalia@sourcegraph.com            │                    198 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ dulani.wallace          │ dulani.wallace@sourcegraph.com          │                    230 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nancy.shah              │ nancy.shah@sourcegraph.com              │                    225 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ megan.standrew          │ megan.standrew@sourcegraph.com          │                    203 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ quinn.hare              │ quinn.hare@sourcegraph.com              │                    188 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ tbliu98                 │ thomas.liu@sourcegraph.com              │                     69 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ vincent.ruijter.appsec1 │ vincent.ruijter+appsec1@sourcegraph.com │                    182 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ sanjeev.sekar           │ sanjeev.sekar@sourcegraph.com           │                    180 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nathan.downs            │ nathan.downs@sourcegraph.com            │                    166 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ alex.isken              │ alex.isken@sourcegraph.com              │                    112 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ nate.maynard            │ nate.maynard@sourcegraph.com            │                     98 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jordan.plahn            │ jordan.plahn@sourcegraph.com            │                     91 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ vdavid                  │ david.veszelovszki@sourcegraph.com      │                     71 │
├─────────────────────────┼─────────────────────────────────────────┼────────────────────────┤
│ jon.galindo             │ jon.galindo@sourcegraph.com             │                     70 │
╰─────────────────────────┴─────────────────────────────────────────┴────────────────────────╯
49 of 209 users were inactive for more than 60 days on https://k8s.sgdev.org.
Do you  wish to proceed with user removal [y/N]: n
Aborting removal

Test plan

This command was tested against our dogfood kubernetes instance and cse-aws docker compose instance.

…been deleted but exist in the aggregated_user_stats table
@DaedalusG DaedalusG changed the title Paginate for through users in users prune - fix bug for soft deleted users users prune - Paginate graphql site users - fix bug for soft deleted users Jan 4, 2023
@DaedalusG
DaedalusG marked this pull request as ready for review January 11, 2023 23:03
Comment thread cmd/src/users.go
Comment thread cmd/src/users_prune.go
Comment thread cmd/src/users_prune.go
Comment thread cmd/src/users_prune.go
Comment thread cmd/src/users_prune.go Outdated
Comment thread cmd/src/users_prune.go Outdated
@DaedalusG
DaedalusG requested a review from mrnugget January 13, 2023 08:25
Comment thread cmd/src/users_prune.go Outdated
@DaedalusG
DaedalusG enabled auto-merge (squash) January 14, 2023 07:52
@DaedalusG
DaedalusG merged commit 2a63033 into main Jan 16, 2023
@DaedalusG
DaedalusG deleted the paginate-users-prune branch January 16, 2023 02:25
jdpleiness pushed a commit that referenced this pull request Jan 19, 2023
…d users (#909)

* fixes bug in which request is made to delete users that have already been deleted but exist in the aggregated_user_stats table

* implemented some pagination -- totalCount of nodes is always 0 -- needing to fix that for break condition

* fixed totalCount issue -- switched DoRaw methods to Do

* remove development print statements; hide users table behind flag; improve comments

* gofmt

* added a few more comments

* style improvements

* better graphQL querry formatting for single use queries
scjohns pushed a commit that referenced this pull request Apr 24, 2023
…d users (#909)

* fixes bug in which request is made to delete users that have already been deleted but exist in the aggregated_user_stats table

* implemented some pagination -- totalCount of nodes is always 0 -- needing to fix that for break condition

* fixed totalCount issue -- switched DoRaw methods to Do

* remove development print statements; hide users table behind flag; improve comments

* gofmt

* added a few more comments

* style improvements

* better graphQL querry formatting for single use queries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants