multi-touch-dev team mailing list archive

[Chase Douglas <chase.douglas@xxxxxxxxxxxxx>]

On Mon, 2010-10-04 at 18:12 -0700, Ping Cheng wrote:
> Hi Chase,
> 
> I didn't have time to look into your Grail/Geis design in-depth. My
> comments are based on what I leant from your presentation at XDS. And
> it is based on the coming XInput 2.1. That is, I assume we have MT
> support in the X server.
> 
> Tell me as soon as I lose you.
> 
> Ping

Thanks for taking a look and responding!

> On 10/4/10, Chase Douglas <chase.douglas@xxxxxxxxxxxxx> wrote:
> > As I've thought some more about the current grail architecture and the
> > future when we have XInput 2.1 with full multitouch support, I realized
> > there may be an issue with our current implementation.
> >
> > Grail currently takes MT input and looks for a subscribed gesture. If
> > there wasn't any, the MT input is discarded (leaving aside single touch
> > emulation). Once XInput 2.1 is available, including when we move gesture
> > recognition to the client side, we'll need to be able to pass through MT
> > events. A wrinkle is that we need to be able to support MT grabbing as
> > well.
> 
> I think most of what you do now in Grail would be handled by XInput
> 2.1. We may still need to keep mtdev to translate type A to type B.
> But that is very much device specific. I think your current Grail or
> the coming XInput 2.1 deals with generic system wide MT events. Right?
> So, once XInput supports MT grabbing, it would be up to the client to
> tell X server that it is grabbing the events or not. Grail/Geis would
> be in the same position as all the other clients, assuming Grail and
> Geis will deal with system wide (or Unity in your term :) MT gestures
> on Ubuntu then.

I'll try to explain where grail fits in detail, and that may help
alleviate any misunderstanding.

Grail is merely a gesture recognition engine. It doesn't know anything
about system-wide vs application gestures. It also doesn't know anything
about grabs. It just takes MT events from XInput 2.1 and attempts to
recognize gestures from them.

The key issue here is ensuring that, when we add MT grab support to X to
properly handle system-wide gestures, grail behaves in a way that allows
the window manager to make the decision on whether to process MT events
itself or release them from its grab so they can be handled by client
applications. I see the following two possible sequences of events,
where the window manager (WM) has grabbed MT events:

1. System-wide gesture is performed
a. A series of MT events are sent by X to the WM and other clients
(flagged as being grabbed by the WM for now)
b. WM passes MT events to grail
c. Grail recognizes a gesture that WM subscribed to
d. Grail sends gesture event to WM
e. WM tells X that it's handling all the touches that comprise the
gesture
f. X tells all the client apps to discard the MT events

2. Application specific gesture or non-gesture MT events are performed
a. A series of MT events are sent by X to the WM and other clients
(flagged as being grabbed by the WM for now)
b. WM passes MT events to grail
c. Grail does not recognize a gesture that WM subscribed to
d. Grail sends MT events back to WM
e. WM tells X to "replay" MT events for the touches to other clients
f. X tells all the client apps they may fully process MT events now

The issue I see with grail right now is that step c in sequence 2 is
indefinite in time. We need to ensure that there's a proper timeout so
the WM can make a decision on the grabbed events.

One last point I'd like to make is that grail/geis is not a client of X.
It's a library used by a client of X to perform gesture recognition.

> > When I thought about MT grabbing, I realized that grabbing really needs
> > to be done on a per-finger basis, and the client needs to decide early
> > on whether to grab the touch through the entire touch, or whether it
> > should allow events through to non-grabbing clients.
> 
> I don't feel per-finger grabbing would work. My view of grabbing works
> like this: when the touch points/contacts are inside an MT enabled
> client, the client grabs the events and translates them into its
> specific events/gestures. The client ungrabs the events when all the
> touch points leave the surface. Next time when MT events coming in,
> grabbing will be decided again base on which client the touch points
> are in. Unity grabs the events if the touch points cover more than one
> client. Same thing happens to Unity. It ungrabs the events only when
> all touch points leave the surface.

I think the proper grab semantics for the purposes of system-wide
gestures is for the WM to have a passive grab on MT devices. This grab
would be activated when a touch begins, and deactivated when a touch
ends. However, each grab would be per-touch, and the WM could choose to
"replay" events for any given touch to other clients. Replaying events
essentially means the client relinquishes the grab for that touch, and
all further events from that touch will not be grabbed as well.

What you have described above sounds like grabbing without replaying
events. This inhibits the ability for the WM to see MT events, realize
they aren't system-wide gestures, and then release the events so other
clients can handle them.

> > Imagine you perform a two finger pinch gesture and then move directly
> > into a two finger drag. If a client subscribes only to the drag gesture,
> > it will receive gesture events for the drag once grail starts to
> > recognize it. However, this breaks down when we try to involve MT grabs.
> > We would need to wait through an entire touch sequence until a
> > subscribed gesture is found before we can decide whether to handle all
> > the events ourselves or replay them to other clients.
> 
> I don't think Grail or XInput can make the decision. It is the client
> who needs to make the decision. I think most gestures are client
> specific. Grail/Geis can only deal with the system wide generic
> gestures. So, it should not care about if a client is doing a draging
> or pinching. It only knows that the client is grabbing the events. If
> the client doesn't support MT or doesn't grab the MT events by some
> reason, Unity kicks in to translate those MT into system wide
> gestures. Unity comes into play the last. It is a system wide default,
> which could be different for different distributions, I guess.

If I read this correctly, I think you understand grail/geis as some
system-wide service that recognizes only system-wide gestures.
Grail/geis are actually libraries that will be incorporated into each
client application that wishes to utilize gesture recognition. This
includes the WM for system-wide gestures as well as client applications
for application gestures like scrolling and zooming.

Unity actually is the first client to receive MT events and attempt to
process them. Only after it has been determined that the events do not
comprise a system-wide gesture are the events then replayed to be
handled by other clients.

> > What I think we'll need to do is modify grail so that it recognizes a
> > subscribed gesture very soon after a touch begins. If no gesture was
> > recognized, the touch needs to be passed through as normal MT events for
> > the remaining duration of the touch. This allows a grabbing client to
> > make a decision on whether to replay events for the touch to
> > non-grabbing clients. This also means grail can't recognize gestures
> > based on touches that have already been disposed to pass-through status.
> 
> I think I've covered this part already.

Thanks a lot for reviewing this. It seems there's some disconnect
between us on how grail/geis fit into the picture, but I hope that
through this discussion we can clear up any issues and plug any holes in
the theory :).

-- Chase