[QUIC] Abort on cancellation throws QUIC_STATUS_INVALID_PARAMETER #73688

Activity

1. CarnaViire

   added

   area-System.Net.Quic

   on Aug 10, 2022
3. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Aug 10, 2022
4. karelz

   added this to the 7.0.0 milestone on Aug 11, 2022
5. karelz

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Aug 11, 2022
6. karelz commented on Aug 11, 2022

   

   karelz

   on Aug 11, 2022

   Last edited by karelz

   Member

   More actions

   Triage: We should look into that.
   @CarnaViire is running stress locally to get a repro, but couldn't reproduce it just yet. We might punt to 8.0 it if we won't be able to find anything useful.

   Putting it into Low Priority bucket as we cannot reproduce it locally and it has only rare hit (1x per week so far).
7. karelz

   modified the milestones: 7.0.0, 8.0.0 on Aug 12, 2022
8. karelz commented on Aug 12, 2022

   

   karelz

   on Aug 12, 2022

   Member

   More actions

   Moving it to 8.0 as we were unable to reproduce it and make it actionable. Impact seems to be low as well.
9. rzikm commented on Aug 15, 2022

   

   rzikm

   on Aug 15, 2022

   Member

   More actions

   Reproduced in #73817 at https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-73817-merge-3f1c7c257c234ddebf/System.Net.Quic.Functional.Tests/1/console.8d88137e.log?helixlogtype=result

   Unhandled exception. System.AggregateException: One or more errors occurred. (An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER)
    ---> System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
      at System.Net.Quic.ThrowHelper.ThrowIfMsQuicError(Int32 status, String message) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ThrowHelper.cs:line 123
      at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 422
      at System.Net.Quic.QuicStream.<>c.<.ctor>b__23_0(Object target) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 75
      at System.Net.Quic.ResettableValueTaskSource.<>c.<TryGetValueTask>b__12_0(Object obj, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 85
      at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 874
      at System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 1104
      at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
      --- End of inner exception stack trace ---
      at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
      at System.Threading.CancellationTokenSource.TimerCallback(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 35
      at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 703
      at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 666
      at System.Threading.TimerQueue.FireNextTimers() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 330
      at System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/TimerQueue.Portable.cs:line 139
      at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 984
      at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77
      at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
   

   My suspicion is that we may have some race which could result in calling StreamShutdown twice.
10. carlossanlop commented on Aug 18, 2022

    

    carlossanlop

    on Aug 18, 2022 · Hidden as duplicate

    show commentMore actions
11. CarnaViire commented on Aug 18, 2022

    

    CarnaViire

    on Aug 18, 2022

    MemberAuthor

    More actions

    I suspect that the reason is not a race but a native heap corruption, note the test suite also crashed with SIGABRT as in #72696
12. hoyosjs commented on Aug 22, 2022

    

    hoyosjs

    on Aug 22, 2022 · Hidden as duplicate

    show commentMore actions
13. carlossanlop commented on Aug 23, 2022

    

    carlossanlop

    on Aug 23, 2022 · Hidden as duplicate

    show commentMore actions
14. hoyosjs

    added

    Known Build ErrorUse this to report build issues in the .NET Helix tabUse this to report build issues in the .NET Helix tab

    on Aug 23, 2022
15. 13 remaining items

    Load more

16. CarnaViire commented on Aug 26, 2022

    

    CarnaViire

    on Aug 26, 2022

    Last edited by CarnaViire

    MemberAuthor

    More actions

    Yep, I know, just reversing the order is not enough, the change would be extremely tricky, there's also a thing that we don't want to lose OCE, and just calling Abort would result in a different exception being set. That's why I'm not suggesting changing that now, merely highlighting that I believe there's a problem and we might want to explore potential solutions.
17. ManickaP commented on Aug 26, 2022

    

    ManickaP

    on Aug 26, 2022

    Last edited by ManickaP

    Member

    More actions

    You cannot abort before setting OCE, you could end up with a different exception than expected OCE, breaking the stream conformance tests thus an expected stream behavior. And as far as I understand we didn't abort at all in 6.0.
    Also we have an open issue to consider making cancellation "soft": #72607 so we can discuss cancellation behavior there.
18. ManickaP commented on Aug 26, 2022

    

    ManickaP

    on Aug 26, 2022

    Member

    More actions

    We would have to change the implementation of ResettableValueTaskSource and move the call to CancellationAction inside the lock in TryComplete, thoughts, @ManickaP?

    That could work, it would be ugly as hell though 🤣
19. rzikm commented on Aug 26, 2022

    

    rzikm

    on Aug 26, 2022

    Member

    More actions

    You cannot abort before setting OCE, you could end up with a different exception than expected OCE

    That is something that is possible to fix (e.g. having internal overload of Abort which takes an exception to use) 😈

    👎React with 👎1Reacted by Marie Píchová
20. ManickaP commented on Aug 26, 2022

    

    ManickaP

    on Aug 26, 2022

    Member

    More actions

    The thing with setting up oce and than aborting is that you set up 2 exceptions in a row, since you want to return OCE first and for all following calls to return StreamAborted (or whatever). And that's a thing that ResettableValueTaskSource is made to do. So moving this logic outside defeats the purpose and it can all be redone in a different way.

    👍React with 👍1Reacted by Radek Zikmund
21. carlossanlop commented on Aug 30, 2022

    

    carlossanlop

    on Aug 30, 2022 · Hidden as duplicate

    show commentMore actions
22. rzikm

    closed this as completedin #74634on Sep 7, 2022
23. ghost removed

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Sep 7, 2022
24. rzikm commented on Sep 7, 2022

    

    rzikm

    on Sep 7, 2022

    Member

    More actions

    Reopening for servicing 7.0
25. rzikm

    reopened this on Sep 7, 2022
26. ghost added

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Sep 7, 2022
27. ghost removed

    in-prThere is an active PR which will close this issue when it is mergedThere is an active PR which will close this issue when it is merged

    on Sep 8, 2022
28. karelz commented on Sep 8, 2022

    

    karelz

    on Sep 8, 2022

    Member

    More actions

    Fixed in 8.0 (main) in PR #74634 and in 7.0 (RC2) in PR #75179
29. karelz

    closed this as completedon Sep 8, 2022
30. ghost locked as resolved and limited conversation to collaborators on Oct 8, 2022