multi-touch-dev team mailing list archive

["Stephen M. Webb" <stephen.webb@xxxxxxxxxxxxx>]

On Mon, 2010-10-11 at 05:07 -0400, Rafi Rubin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 10/08/10 12:49, Stephen M. Webb wrote:
> > On Fri, 2010-10-08 at 09:40 -0600, Duncan McGreggor wrote:
> >>
> >> I believe Stephen already has some changes he wants to make to this
> >> proposed spec, so expect updates :-)
> > 
> > I'm thinking of radical changes, in fact.  This came out of the XDS
> > decision to move gesture recognition into the client side.
> > 
> > Before I go to far, I want to give a brief description of what I think
> > the 2.0 API should be like to see if this jives with everyone.
> > 
> > Geis 2.0 will have two ends:  an input end and a recognition end.
> > 
> > The application or toolkit will receive input events from, well,
> > wherever.  XI2.1 in the case of X11, or native inputs in the case of a
> > framebuffer-based installation (geis should not depend in any way on X).
> > To that end, the application or toolkit will need to get an input cooker
> > from geis, one per input device I imagine, and feed the raw input events
> > into the cooker in its event loop.  The cooker transforms the raw input
> > events into internal geis input events, and may require further inputs
> > such as coordinate mappings depending on the requirements.
> 
> Just to be clear, are you talking about 1 central geis process per input device,
> or one per device for each client?

I'm imagining this as a client-side library with the potential for
multiple recognizer instances per client (to support different gesture
sequences in subregions, for example).  Having the recognition in an
external process not part of the input data flow makes me hesitate
because of latency and race condition issues.  I think the problems
would outweigh the advantages to that approach.

I think each recognizer instance should be able to handle input from one
or more input devices.  Because the application handles the processing
of input devices, if it wants certain gesture sequences handled only for
certain input devices, it will have to do the filtering.  If it knows it
needs gestures from only those devices, it should already be aware of
the specific devices so this should be no loss.

> > The application or toolkit will then have to poll the geis event loop.
> > This drives outstanding cooked input events through the recognizer and
> > results in zero or more events being output.  The possible events
> > include (1) preview events -- the so-called tentative events, (2)
> > gesture events, just like in geis 1.0, (3) touch events, for touches
> > that are not recognized as part of a gesture, and (4) user-defined
> > events, discussed below.  The application or toolkit can do what it
> > likes with these events.
> 
> So if my understanding is correct, you prepare "proto-gestures" in the core geis
> space, and then process that into the more concrete events in the client.

No.  Raw input from an input source (which will be XI2.1, first pass)
arrives in the client event loop, gets sent to a cooker to be
transformed into a geis input format (this is akin to the "transform
layer" of Florian Echtler) and sent into the recognizer chain.  My
original thoughts were to have fixed recognizer (grail) recognize
gesture primitives (touch, translate, scale, rotate) and pass these on
to the optional programmable recognizer, and the touch and gesture
events are sent back into the client's event loop.

Everything is performed in the client application space.

> A good comparison, I wouldn't be too surprised to hear that someone actually
> does play with a gpu to accelerate gesture recognition :)


Hehe, I thought of that but gesture recognition is so cheap compared
with processing millions of vertexes that I believe the overhead of
memory space transfers and context switching would overwhelm any
benefits.  I could be wrong.

> As long as you don't need to drastically increase the effort for a client trying
> to get the functionality it had with 1.0 in the new api, and you see a clear
> path to implementation then I think this is quite sane.

I think the effort to obtain basic functionality with this plan will
increase, in that the client will need to process input events more
explicitly.  I think it is a worthwhile tradeoff in that no callback
functions or region specifications are necessary.

-- 
Stephen M. Webb <stephen.webb@xxxxxxxxxxxxx>
Canonical Ltd.