multi-touch-dev team mailing list archive

[Chase Douglas <chase.douglas@xxxxxxxxxxxxx>]

On Wed, 2010-10-13 at 17:16 +0200, Henrik Rydberg wrote:
> On 10/13/2010 04:14 PM, Chase Douglas wrote:
> [...]
> >>> If you went this route, where toolkits and apps listen directly to the
> 
> >>> kernel and are assumed to propagate events between them appropriately,
> >>> you'll have issues once you start mixing in legacy apps and toolkits
> >>> that listen for this data from X.
> >>
> >> We would of course have a single point of event propagation, so "between them"
> >> would not enter the equation. Since there is no gesture data from X, there is no
> >> mixing. And there are no applications listening to MT events from X either,
> >> since it does not exist yet. Thus, there are no real issues here, except in what
> >> direction we want to go.
> > 
> > Ok, in this case I'm having a hard time seeing how this is different
> > than putting the cross-"windows" MT propagation logic in the X server
> > and having gestures and widget MT propagation client side. We can't put
> > gesture recognition anywhere other than in each individual applications
> > because each application may be written with a different toolkit. The
> > recognition regions are only known within the process space of
> > individual clients.
> 
> 
> This is under the assumption that we don't add regions to geis, or any other way
> of obtaining windowing information from the toolkits.

Correct. I'll try to answer the unstated question of whether we should
do this with a single geis daemon instance.

Imagine you've got a window with many widgets. One of them is a
scrollable region, so it subscribes to gestures and provides the region
in screen coordinates.

Now the window is moved on screen. Should the toolkit have to update
geis with the new location? If so, it sounds racy. The WM could infer
the new position. However, what happens when the window is resized? The
new area of the scrollable widget is toolkit and application specific,
so the toolkit will need to inform geis of the new region. Again, I see
things getting racy during a resize operation. It also adds a bunch of
round trip messages and potential for UI delay. There may also be
contention for the geis daemon between clients.

In short, I think it could be done this way, but I don't think it's
optimal.

> >>> Also, in order to perform proper propagation between "windows", each
> >>> application will need to query the window stack on every new input event
> >>> to be sure it is allowed to handle the input.
> >>
> >>
> >> Definitely not. The point is that the process doing propagation can have the
> >> full picture because it is placed where all information is available.
> > 
> > The X process space is the only place where window position information
> > is available internally.
> 
> 
> We went half-way towards extracting that information once already, so we know it
> is a bit cumbersome, but possible.

It's possible, but I think it would be racy again. I also worry about
interactions with X input grabbing (not just XI 2.1 MT grabbing, but
even normal ST grabbing). It doesn't strike me as a better or more
elegant solution.

> >> It is clear that the part still unsolved has to do with multi-user input. I
> >> include the question whether a gesture should in fact be seen as a set of
> >> individual fingers. Combinatorics, projections, greedy versus global... I think
> >> we need a handle on this problem before deciding what direction to take.
> > 
> > Certainly skepticism is necessary as we formulate a solution, so thanks
> > for trying to highlight issues. In my mind I don't see any real issues
> > though. Do you have any scenarios where the proposed mechanisms break
> > down?
> 
> 
> Good point. Would you mind going through, step-by-step, the case where we start
> with two fingers inside a "window", scroll a bit, then add another finger in a
> different window, and perform a three-finger pinch?

There's a few context questions we need to ask first (I'm trying to get
at what your asking, maybe I'm going off in the wrong direction though):

1. Is a three finger pinch a system-wide gesture? I'll assume yes. Are
two finger gestures system-wide gestures? I'll assume no.

2. Should a two-to-three finger movement that is not immediate, by which
I mean this isn't a three finger movement from the start where the first
couple of device touch frames caught only two fingers, trigger a three
finger gesture? I don't believe we've explicitly answered this anywhere
yet, but I think not. A three finger gesture must be initiated with
three fingers within a short period of time on the order of 10s of
milliseconds (I'm leaving aside the case of gesture continuation).

Given these assumptions, I think the expected result is that the
scrollable region continues to scroll according to the two fingers over
it, and the single finger in the other window does whatever it should in
that window.

>From the plumbing perspective I see the following sequence of events:

1. Two fingers touch down in scrollable region
2. MT data passed to WM through XI 2.1 passive grab
3. MT data also passed to app toolkit, but with "not-for-you-yet" flag
4. WM and toolkit pass data to their own instances of geis/grail
5. WM grail times out waiting for three fingers
6. WM gets MT data back, realizes there's no gesture, "replays" touches
7. In parallel, app toolkit gets two finger drag gesture from geis/grail
8. App toolkit gets event saying MT events "are for you now"
9. App toolkit starts scrolling
10. Third finger goes down in another window
11. Steps 2-6 and 8 repeated for this touch
12. App toolkit for this other window handles touch as appropriate

Note that after the WM replays any touches, no more events generated by
those touches are sent to it. It will only get events from new touches
to the screen.

Also, during the "not-for-you-yet" phase, applications can provide UI
feedback or do anything else with the data as long as they appropriately
handle both cases where the touches end up replayed or consumed by the
grabbing client.

I hope that covers the use case and I'm not forgetting anything :).
Thanks for asking about this scenario, I've been meaning to post a
detailed event sequence but just haven't gotten around to it yet.

-- Chase