View managers should be able to handle/capture pointer events

Eric Rozell (@rozele)

copying from Eric Rozell (@rozele)'s PR here so we can have a design discussion: #8495

Currently, there is no way for ViewManagers to capture the pointer to perform it's own pointer event handling. This change introduces a new interface, IViewManagerWithPointerEvents, so third party modules can be notified when the ReactRootView receives pointer events. The behavior is as follows:

If IViewManagerWithPointerEvents returns true in OnPointerPressed, TouchEventHandler will not capture or start tracking the pointer for React Native touch events.

If IViewManagerWithPointerEvents returns true in OnPointerMoved, TouchEventHandler will release capture of the pointer if there is an active pointer tracked for a React Native gesture. Open question: how should onMouseEnter / onMouseLeave events be handled if the IViewManagerWithPointerEvents is tracking the pointer?

If IViewManagerWithPointerEvents returns true in OnPointerReleased, TouchEventHandler will send a cancellation event
to React Native rather than an end event.

Here are a couple scenarios for how a custom view manager might use this:

A native pressable feature that prevents RN responders: return true in OnPointerPressed or OnPointerReleased
A native draggable that is only triggered after a threshold is exceeded: start tracking pointer movement in OnPointerPressed, check threshold on each OnPointerMoved event, if exceeded, return true.

Chris Glein (@chrisglein)

The main motivation for why we should create something that allows cooperation between view managers and the TouchEventHandler is that TouchEventHandler captures the pointer in PointerPressed. This prevents child controls that may want to subscribe to PointerMoved events, or capture the pointer after the PointerPressed event from being able to do so.

One caveat here is that XAML controls like TextBlock and ScrollViewer successfully capture the pointer after the TouchEventHandler on the ReactRootControl (forcing a "PointerCaptureLost" event in the TouchEventHandler). Perhaps these controls use win32 APIs under the hood and tree traversal to release parents, but it's opaque because it's a XAML implementation detail.

So, given this, what options do we have?

Do not capture the pointer in TouchEventHandler

This is a bad option. It's quite useful that controls like TextBlock and ScrollViewer release pointer captures of their ancestor views, canceling React touch events when scrolling or highlighting selectable text.

Force all view managers to follow whatever pattern TextBlock and ScrollViewer use under the hood.

If there was a trivial way to subscribe to PointerPressed|Moved|Released while a parent has captured the pointer, then we could simply do that, and release the ReactRootView by walking the tree of parents until you find the one that has captured the pointer. Hopefully someone from the XAML team can comment on this option (cc Chris Glein (@chrisglein), Steven Moyes (@stmoy)).

Add an cooperative interface for view managers to participate in pointer capture behavior (as in Adds hook for view managers to capture pointer events #8495)

If there is no developer-friendly way to imitate the behavior of TextBlock and ScrollViewer which can release ancestor pointer captures, then a good alternative is to create a cooperative interface that allows arbitrary view managers to prevent the TouchEventHandler from capturing the pointer and/or emitting React Native pointer events. One possible solution is given in #8495.

The main concern with option 3 is performance. Tracking PointerMoved events on the root view already causes quite a bit of CPU usage. The CPU usage comes from:
a) Finding the React tag from the OriginalSource (TagFromOriginalSource), which occasionally uses VisualTreeHelper::GetParent to walk the tree until it finds the first parent with a React tag
b) Text hit testing, which does a bounds check on all descendant Runs of a TextBlock control (improved slightly in #7553 with ShadowNode state for pruning branches that do not have pressable props).
c) Walking from OriginalSource to ReactRootView on every pointer move for onMouseEnter and onMouseLeave events.

Having to send callback messages for each PointerMoved event to all ViewManagers in the branch from the ReactRootView to the OriginalSource would not make this situation any better.

Additionally, if we want to implement "box-only" correctly (as in #8500), we need to walk from OriginalSource to ReactRootView on each pointer move to determine if there are any React ancestors of the OriginalSource with pointerEvents="box-only" set.

However, I believe we can consolidate the tree walks for "box-none", "onMouseEnter/Leave", and something like IViewManagerWithPointerEvents to limit the overhead to not much more than we already experience in RNW v0.65.

We can likely also sequence the events in such a way that we do CPU work only when necessary. E.g., in TagFromOriginalSource, we can find the first React view, then, only if no ancestor has "box-only", further hit test the target (e.g., for Text hit testing).

Another potential perf optimization is to maintain state at each shadow node to signal whether a particular view is a descendant of a view with prop pointerEvents="box-only", a descendant of a view with onMouseEnter/onMouseLeave, or a descendant of a view that implements the approach in #8495, so we could avoid tree walks all together (but this certainly adds a lot of complexity, and slows "rendering" time to speed up pointer events).

That being said, before we overengineer perf optimizations, we should just track the CPU impact of #8500 and #8495 and decide whether we should go forward with a simple approach first and optimize later, or if optimization is urgent.

If IViewManagerWithPointerEvents returns true in OnPointerMoved, TouchEventHandler will release capture of the pointer if there is an active pointer tracked for a React Native gesture.

API nit: returning true/false makes the behavior ambiguous from just a method signature.

Some solutions I could see to this:

Something akin to https://developer.mozilla.org/en-US/docs/Web/API/Event/stopPropagation, where handler can interact with event object.
If forcing a choice is desirable (requiring return), an enum like HitResult::NoCapture

This change would give precedence to native hit test compared to JS-hit test of react event against their component tree. I'm wondering if there are cases where native components are ineligible for hit test due to JS-side state, or other cases where JS would need to control capture at a coordinate where something with native capture is present.

I like the idea of having a wrapper class around PointerRoutedEventArgs. I also considered piggy backing off the Handled() property on PointerRoutedEventArgs, but this boolean value is probably just as ambiguous as a boolean return value. I also just thought of a couple additional use cases:

In #7813 - we added behavior to prevent the OnPointerCaptureLost callback from canceling the React pointer in certain conditions. Specifically, if the user clicks on selectable text but does not change selection, we do not want the PointerCaptureLost event to cancel the React pointer event.

In #7553 - we added a more involved hit testing algorithm for Text. In #8493, we added a virtual method to ViewManagerBase so any view could refine the React target view for hit testing. We can actually implement this using just this IViewManagerWithPointerEvents interface by supplying a Target setter on the args passed to the view manager.

So, here are a handful of cases that we want to enable:

Prevent capture on PointerPressed
Release capture on PointerMoved
Cancel React event on PointerReleased
Prevent cancellation on PointerCaptureLost
Refine React target view on any pointer event

Something I haven't thought through entirely is what control different view managers need to interoperate. It may be non-trivial to build state machines in view managers to know when it was responsible for releasing capture of TouchEventHandler, though arguably no more difficult than dealing with routed pointer events in the first place. For example, if a child view releases pointer capture, the parent view manager will need to retain enough state that it knows it was not the one that released capture and does not respond to the OnPointerCaptureLost callback.

Nick Gerleman (@NickGerleman)

So, building on your Web Event API reference Nick Gerleman (@NickGerleman), if we had a wrapper like:

class PointerEventCallbackArgs {
  // Do not fire React event
  void PreventDefault();
  
  // If TouchEventHandler has captured the pointer, release it or prevent capture in PointerPressed callback
  void ReleaseCapture();
  
  // Do not fire callbacks for parent view, and do not fire React event
  void StopPropagation();
  
  // Refine the React target
  object Target { get; set; }
  
  PointerRoutedEventArgs Args { get; }
}

Rather than having implied behavior for when TouchEventHandler should release pointer capture, it's probably better to give explicit control of when this occurs, thus the ReleaseCapture() method which doesn't show up in the Web Event API.

Then for each scenario:

Prevent capture on PointerPressed: call ReleaseCapture() (optionally we could also force calling PreventDefault())
Release capture on PointerMoved: call ReleaseCapture()
Cancel React event on PointerReleased: call ReleaseCapture() (optionally we could also force calling PreventDefault())
Prevent cancellation on PointerCaptureLost: call PreventDefault()
Set the Target property on the React tag to the refined view for each event.

So, for the scenarios described, we don't really require StopPropagation(), but it might be worth adding to give descendant view managers control over what callbacks their ancestors receive.

One thing that's a bit problematic is when you force the ReactRootView to release capture of the pointer, it will revert to onMouseEnter / onMouseLeave / onMouseMove behaviors for PointerMoved events. If a ViewManager wants to take complete control over the pointer events, it will likely need to do the following:

Call ReleaseCapture at the point that it wants to start handling events
If it wants to continue handling events during a gesture (even after the mouse leaves the view box), it will need to capture the pointer
It will need to set Handled(true) on the PointerRoutedEventArgs to ensure that the ReactRootView does not receive events to trigger onMouseEnter / onMouseLeave.

Can we build in a way to defer releasing capture until gesture end?

Though as a generality I think this proposal makes sense. It should also have less red tape, since our native APIs are fully our own.

Can we build in a way to defer releasing capture until gesture end?

Is there a specific reason for this? Or just to counteract the issue I mentioned above where we switch back to onMouse* events after canceling the RN gesture?

Though as a generality I think this proposal makes sense. It should also have less red tape, since our native APIs are fully our own.

Indeed - I think the proposal here should be general enough to handle most scenarios, even if the developer experience can be a bit tricky to nail when trying to defer to a native view.

Is there a specific reason for this? Or just to counteract the issue I mentioned above where we switch back to onMouse* events after canceling the RN gesture?

Trying to understand semantics for the below scenario. I'm not super familiar with how XAML input works, or current RNW stack though, so this may already be handled.

For the flow:

User starts touch gesture
On pointer starts, VM decides to handle
VM calls preventDefault to avoid passing to TouchEventHandler
Pointer move event dispatched
VM does not handle

Do we end up dispatching touchMove + touchEnd without a start to the JS-side?

Same for things like pointerup without pointerdown. It feels like some gestures might need to be atomic. Not sure how web handles it, but IIRC RN JS-side gets fussy if sent a partial event stream.

Do we end up dispatching touchMove + touchEnd without a start to the JS-side?

We will explicitly prevent this from happening. We can do this by making the semantics of PreventDefault equivalent to ReleaseCapture in the PointerPressed event callback. PointerPressed creates a tracking object that is required for onTouchMove and onTouchEnd events to be emitted.

Same for things like pointerup without pointerdown. It feels like some gestures might need to be atomic.

Yes - setting the default semantics for PreventDefault == ReleaseCapture on PointerPressed will ensure that events are only received if an onTouchStart event is emitted.

If we get a PointerMoved event when we haven't captured / started tracking a PointerPressed event, we emit onMouseEnter / onMouseLeave events. See:
https://github.com/microsoft/react-native-windows/blob/main/vnext/Microsoft.ReactNative/Views/TouchEventHandler.cpp#L163-L169

If we get a PointerCanceled/PointerReleased/PointerCaptureLost event when we haven't started tracking PointerPressed, we ignore. See:
https://github.com/microsoft/react-native-windows/blob/main/vnext/Microsoft.ReactNative/Views/TouchEventHandler.cpp#L177-L179

Thus, if a view manager calls PreventDefault or ReleaseCapture during an OnPointerEvent callback for PointerEventKind::Start, then that view manager should be prepared to either:
a) Call PreventDefault in subsequent OnPointerEvent callbacks for PointerEventKind::Move, or...
b) Call Handled on the PointerRoutedEventArgs so the event does not bubble to the root view

If the view manager fails to do this, then we could get into a situation where the TouchEventHandler starts emitting onMouseEnter/onMouseLeave events while a view manager is presumably tracking the pointer itself.

Okay - discovering some new things about pointer captures. As it turns out, pointers can only be captured during the PointerPressed callback. So, ReleaseCapture is sort of a pointless API, we should only need "PreventDefault". What we also need to support our current use case is the ability to allow a child RN view to capture the pointer, but still allow RN to continue emitting events. So I'm thinking something like:

class ReactPointerEventArgs {
  // Do not fire React event
  void PreventDefault();
  
  // Fire React events even if pointer is not captured successfully
  void AllowUncaptured();
    
  // Refine the React target
  object Target { get; set; }
  
  PointerRoutedEventArgs Args { get; }
}

The semantics would be the following:

PointerPressed: If PreventDefault() is called, do not start tracking the pointer in RN. If PreventDefault() is not called and the call to CapturePointer returns false, only track the pointer if AllowUncaptured is also called. Otherwise, unchanged.
PointerCaptureLost: If AllowUncaptured() is called, do not emit a pointer canceled event. Otherwise, unchanged.
PointerMoved: If PreventDefault() is called, do not emit pointer move events. If PreventDefault() is not called and the pointer is not captured, stop tracking the pointer in RN if AllowUncaptured is not called (by sending a pointer canceled event). Otherwise, unchanged.
PointerCanceled: Unchanged.
PointerReleased: If PreventDefault() is called, emit a pointer canceled event in RN rather than a pointer end event. If the pointer is not captured and AllowUncaptured is not called, emit a pointer canceled event in RN.

Here it is in pseudocode:

onPointerPressed(view, args) {
  if (!args.defaultPrevented && 
      (view.pointerCaptures.push(args.pointer) || args.uncapturedAllowed)) {
    trackedPointers.push(args.pointer)
    emitStart();
  }
}

onPointerMoved(view, args) {
  if (trackedPointers.contains(args.pointer)) {
    if (!view.pointerCaptures.contains(args.pointer) && !args.uncapturedAllowed) {
      trackedPointers.remove(args.pointer);
      emitCancel();
    } else if (!args.defaultPrevented) {
      emitMove();
    }
  }
}

onPointerCaptureLost(view, args) {
  if (!args.uncapturedAllowed) {
    onPointerConcluded("cancel", view, args);
  }
}

onPointerCancel(view, args) {
  onPointerConcluded("cancel", view, args);
}

onPointerReleased(view, args) {
  if (args.defaultPrevented ||
      (!view.pointerCaptures.contains(args.pointer) && !args.uncapturedAllowed)) {
    onPointerConcluded("cancel", view, args);
  } else {
    onPointerConcluded("end", view, args);
  }
}

onPointerConcluded(eventType, view, args) {
  if (trackedPointers.contains(args.pointer)) {
    trackedPointers.remove(args.pointer);
    view.pointerCaptures.remove(args.pointer);
    if (eventType == "cancel") {
      emitCancel();
    } else if (eventType == "end") {
      emitEnd();
    }
  } 
}

Navigation Menu

Repository navigation

View managers should be able to handle/capture pointer events #8499

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

View managers should be able to handle/capture pointer events #8499

Description

Activity

rozele commented on Aug 28, 2021

rozele commented on Aug 28, 2021

rozele commented on Aug 28, 2021

NickGerleman commented on Aug 28, 2021

NickGerleman commented on Aug 28, 2021

rozele commented on Aug 28, 2021

rozele commented on Aug 28, 2021

rozele commented on Aug 31, 2021

NickGerleman commented on Aug 31, 2021

NickGerleman commented on Aug 31, 2021

rozele commented on Aug 31, 2021

NickGerleman commented on Sep 2, 2021

NickGerleman commented on Sep 2, 2021

rozele commented on Sep 2, 2021

rozele commented on Sep 2, 2021

rozele commented on Sep 2, 2021

rozele commented on Sep 8, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions