multi-touch-dev team mailing list archive

[Peter Hutterer <peter.hutterer@xxxxxxxxx>]

On Fri, Oct 15, 2010 at 07:56:21PM +0200, Henrik Rydberg wrote:
> On 10/15/2010 06:12 PM, Chase Douglas wrote:
> 
> > On Fri, 2010-10-15 at 17:23 +0200, Henrik Rydberg wrote:
> >> On 10/14/2010 09:39 PM, Chase Douglas wrote:
> >>> On Wed, 2010-10-13 at 20:43 +0200, Henrik Rydberg wrote:
> >>>> On 10/13/2010 06:03 PM, Chase Douglas wrote:
> >>>>>> Good point. Would you mind going through, step-by-step, the case where we start
> >>>>>> with two fingers inside a "window", scroll a bit, then add another finger in a
> >>>>>> different window, and perform a three-finger pinch?
> >>>>>
> >>>>> There's a few context questions we need to ask first (I'm trying to get
> >>>>> at what your asking, maybe I'm going off in the wrong direction though):
> >>>>>
> >>>>> 1. Is a three finger pinch a system-wide gesture? I'll assume yes. Are
> >>>>> two finger gestures system-wide gestures? I'll assume no.
> >>>>>
> >>>>> 2. Should a two-to-three finger movement that is not immediate, by which
> >>>>> I mean this isn't a three finger movement from the start where the first
> >>>>> couple of device touch frames caught only two fingers, trigger a three
> >>>>> finger gesture? I don't believe we've explicitly answered this anywhere
> >>>>> yet, but I think not. A three finger gesture must be initiated with
> >>>>> three fingers within a short period of time on the order of 10s of
> >>>>> milliseconds (I'm leaving aside the case of gesture continuation).
> >>>>
> >>>>
> >>>> I tend to agree, although that is all about policy. I was intending this example
> >>>> to trigger a global gesture, since it makes the example more difficult.
> >>>>
> >>>>> Given these assumptions, I think the expected result is that the
> >>>>> scrollable region continues to scroll according to the two fingers over
> >>>>> it, and the single finger in the other window does whatever it should in
> >>>>> that window.
> >>>>
> >>>>
> >>>> Right, that is the easy path. So what about the one which triggers a global gesture?
> >>>
> >>> I'm not sure we've defined a mechanism that would work in that case.
> >>> What you would need is the ability to peek at all touch events
> >>> individually and pick and choose which ones you want to keep and which
> >>> you want to replay. The reason this gets dicey is what does the client
> >>> do? Imagine the following:
> >>>
> >>> 1. Touch begins
> >>> 2. WM replays touch events one at a time
> >>> 3. Application gets touch begin, touch motion events
> >>> 4. WM consumes some events
> >>> 5. WM replays more events
> >>> 6. WM consumes events
> >>> 7. Touch ends
> >>>
> >>> Issues:
> >>>
> >>> * Does application get touch begin events when touch physically begins
> >>> if WM starts consuming all events? Or does it only receive a begin event
> >>> once the first replayed event is sent?
> >>> * Is application notified when WM starts consuming events again?
> >>> * What happens when events are replayed in step 5? It may seem that the
> >>> touch moved from one are of the screen to another instantaneously when
> >>> in reality the touches in between were consumed by the WM.
> >>> * If the WM consumes the touch end, does the application then receive
> >>> the touch end?
> >>>
> >>> I think this use case is opposite to good UI design, and I think the
> >>> plumbing to make it possible opens up a big can of worms. X doesn't
> >>> support any replay mechanism like this for even legacy pointer events.
> >>> We seem to all agree that the original example bore the correct outcome,
> >>> so I'm hesitant to continue down this path until we find a scenario that
> >>> would require it.
> >>>
> >>> There's also nothing that prevents us from adding support for this in
> >>> the future if we find we need it.
> >>
> >>
> >> Thanks for showing the implications, Chase. I think it illustrates well that MT
> >> and Gestures cannot be separated.
> >>
> >> If we think of MT events as separate single-finger drags, MT becomes a subset of
> >> gestures. In grail, a N-finger gesture overrides all other gestures that can be
> >> performed with M < N fingers. This includes one-finger drags. Going back to your
> >> example, the natural interpretation is that when the three-finger gesture
> >> starts, the two single-finger drags end. Perhaps if XI2.1 had the ability to
> >> chop a touch up into pieces during replay, the problem could be solved.
> > 
> > I'm not sure I follow. Above, I noted all the issues with chopping up
> > touches during replay. That doesn't seem to be a reasonable solution to
> > me, at least not without more details if you envision some solution to
> > the issues.
> 
> 
> Spot on, some more details are needed here. But contrary to what you are
> writing, I think this is solvable, and a good idea.
> 
> >> This thread, as well as talking some more with Peter, makes me think we are
> >> pretty close. The things I see we need to do is:
> > 
> > Hrm, based on my understanding of things, I think you are venturing
> > further apart from me, and what I thought Peter was thinking too :).
> 
> 
> I have been a lot further away than this ;-)
> 
>  
> >> 1. Add the ability to limit finger grouping to areas in grail.
> >>
> >> The separation into finger groups is already there, but it is only based on the
> >> incoming data. Adding the ability to split the fingers into groups based on
> >> domains could be achieved by an api callback.
> > 
> > Why not just have the toolkit/application pass touches for areas into
> > grail separately? I'm not seeing why we need an api for regions in
> > grail/geis?
> 
> 
> If a single application wants two separate gesture areas, for instance, it makes
> it very easy to setup and handle. I cannot see how that could be bad. What would
> the alternative be, using two instances of a recognizer? More importantly, it
> allows for a set of interactions not easily covered in any other way. The two
> areas could be overlapping for instance.
> 
> The bottom line is that our primary goal is to decode the language of gestures,
> and facilitate their usage in applications.
> 
> > My thought was to have grail heuristically determine touch groupings
> > within a set passed to it by the application. If grail sees a large area
> > and touches are far apart, it can try to group them appropriately.
> > There's no region boundaries though.
> 
> 
> This might very well enter too.
> 
> >> The visible simplifications during natty would be more expressive power to the
> >> application. In particular, the ability to have a dedicated scroll area.
> > 
> > This is more easily done through having the toolkit define a widget for
> > the scroll area and passing MT events in that area to grail. I'm not
> > sure what providing a region to grail would accomplish beyond this.
> 
> 
> The way you describe does not sound easier to me, considering the glue to pass
> the messages on, wait for answers (synchronous or asynchronous?), then capture
> the gesture and do something with it. What sounds simple to me is to be able to
> say "this widget listens to these gestures", and just get the gesture events for it.
> 
> >> It would also enable the possibility to perform all event propagation from within
> >> the same process, although we would not push this scenario at all.
> > 
> > Are you talking about propagation between X "windows"? If so, I thought
> > that was deemed ill advised by myself and Peter.
> 
> 
> Hehe, I think we already covered that there is an interest in non-X applications
> as well.
> 
> >> 2. Add the ability to split touches in XI2.1
> >>
> >> The passive grab mechanism needs to be able to gracefully handle consumption of
> >> touches, such that touch-up and touch-down events are inserted appropriately if
> >> the consumer of a touch changes.
> > 
> > Doesn't this just open up the can of worms I described above?
> 
> 
> Yep. Didn't you just ignore the problem by enforcing a certain policy for all
> gesture applications? ;-)
> 
> 
> >> 3. Skip the tentative events in XI2.1
> >>
> >> From Peter's explanation, it seems the original reason for this was to not run
> >> out of buffer space. Since the semantics is still unclear (just follow this
> >> thread for an example), it is probably best pushed to future expansions.
> > 
> > Without tentative events, how will you provide low latency UI feedback
> > to the user?
> 
> 
> The low-latency does not get better by using passive grabs in the first place,
> which is still one of my biggest concerns. I am willing to play along though,
> since given what we have, it might still be the most viable solution. As far as
> tentative events goes, however, I think they are a bad idea.
> 
> > The semantics of tentative events seems clear to me, what issues do you
> > see? In fact, we haven't really discussed the alternative of queuing up
> > grabbed events for serial replaying, but I think that has even more
> > issues because of event ordering. If you go with tentative events, you
> > can send all the events in the correct order the first time. If you
> > queue up events, you have to worry about how device change events (known
> > elsewhere as DCEs) in the XI 2 spec are handled.
> 
> 
> From what I gather, tentative events are passed during the time when an event is
> neither replayed nor consumed. For gestures, this is exactly the time when
> nothing at all should happen. 

Then don't do anything with the events? That's kinda what you're supposed to
do anyway until you get the OK that the event sequence isn't tentative
anymore.

> If the gesture is accepted, the action will be
> controlled by the entity detecting the gesture, and it is up to that entity to
> make sure some kind of feedback is given. Nota bene, the *gesture* may very well
> be tentative at this stage. If the gesture is cancelled, the real events will be
> replayed and things will continue as normal. Thus, I see no reason at all to
> introduce tentative events.

There's a reasonable technical argument for them - to avoid buffering a
possibly huge number of events in the server. Seriously, the tentative
events, whether they will be added or not, add virtually nothing to the
client-side semantics.

In fact, if you always wait until the "use-it-now event" before doing the
handling of touch events and discard those you get a "discard this sequence"
event for, there is no semantical change at all.
 
Cheers,
  Peter

> Without trying to speak for Florian, I believe he did have similar issues with
> the tentative events. Florian, please correct me if I'm wrong. :-)
> 
> >> So, Gentlemen, what do you think of this?
> > 
> > I feel like I'm really missing something, because most of the thoughts
> > above are opposite to what I've been thinking all along. Maybe I'm just
> > not reading them right?
> 
> 
> That is my hope :-)
> 
> Cheers,
> Henrik