Skip to content

fixed rule manager race condition - #545

Closed
afek854 wants to merge 5 commits into
mainfrom
fix-race-conditions
Closed

afek854 wants to merge 5 commits into
mainfrom
fix-race-conditions

Conversation

@afek854

@afek854 afek854 commented Apr 29, 2025 •

Copy link
Copy Markdown
Contributor

When the node-agent crash while in learning period, it does not update the AP status. (It got stuck as Ready).

  • To reproduce, learning period (2m) -> crash -> status not changing

Explanation:

Container Restart Sequence:
--------------------------

Time      ContainerCallback          monitorContainer         deleteResources         saveProfile
----      ----------------           ---------------          --------------          -----------
t0        AddContainer               
t1                                   Start monitoring
t2                                   SetStatus(Initializing)
t3                                                                                   First save (Initializing)
t4        RemoveContainer
t5                                   ← Gets error signal
t6                                   SetStatus(Completed)
t7                                                                                   Final save (Completed)
t8                                   Exits                     
t9                                                            Deletes resources
t10       AddContainer (new)      
t11                                  Start monitoring (new)
t12                                  SetStatus(Initializing)
t13                                                                                  First save (Initializing)
t14       [RACE CONDITION]: If t9 (deleteResources) happens after t10-t13, it wipes out the tracking data for the new container
t15                                  Ticker fires, but tracking data is gone
t16                                  Cannot transition to Ready

afek854 added 2 commits April 29, 2025 16:31
Signed-off-by: Afek Berger <afekb@armosec.io>
Signed-off-by: Afek Berger <afekb@armosec.io>
@afek854
afek854 marked this pull request as ready for review April 29, 2025 13:39
@matthyx

matthyx commented Apr 29, 2025

Copy link
Copy Markdown
Contributor

good idea, let's continue in this direction

@github-actions

Copy link
Copy Markdown

Summary:

  • License scan: failure
  • Credentials scan: failure
  • Vulnerabilities scan: success
  • Unit test: success
  • Go linting: success

Signed-off-by: Afek Berger <afekb@armosec.io>
@afek854
afek854 force-pushed the fix-race-conditions branch from 6b9952a to 412f56c Compare April 29, 2025 14:10
@github-actions

Copy link
Copy Markdown

Summary:

  • License scan: failure
  • Credentials scan: failure
  • Vulnerabilities scan: success
  • Unit test: success
  • Go linting: success

@matthyx
matthyx force-pushed the fix-race-conditions branch from 9b5c86d to b0d70b4 Compare April 30, 2025 15:57
@github-actions

Copy link
Copy Markdown

Summary:

  • License scan: failure
  • Credentials scan: failure
  • Vulnerabilities scan: success
  • Unit test: success
  • Go linting: success

@matthyx matthyx moved this to WIP in KS PRs tracking May 2, 2025
@afek854
afek854 force-pushed the fix-race-conditions branch from a44ec59 to ab62298 Compare May 4, 2025 07:22
Signed-off-by: Afek Berger <afekb@armosec.io>
@afek854
afek854 force-pushed the fix-race-conditions branch from ab62298 to 6c05669 Compare May 4, 2025 07:33
@github-actions

github-actions Bot commented May 4, 2025

Copy link
Copy Markdown

Summary:

  • License scan: failure
  • Credentials scan: failure
  • Vulnerabilities scan: success
  • Unit test: success
  • Go linting: success

…conditions

Signed-off-by: Afek Berger <afekb@armosec.io>
@afek854
afek854 force-pushed the fix-race-conditions branch from d2250e7 to a900ad3 Compare May 4, 2025 08:12
@github-actions

github-actions Bot commented May 4, 2025

Copy link
Copy Markdown

Summary:

  • License scan: failure
  • Credentials scan: failure
  • Vulnerabilities scan: success
  • Unit test: success
  • Go linting: success

@afek854 afek854 closed this May 4, 2025
@matthyx matthyx moved this from WIP to To Archive in KS PRs tracking May 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants