meadl-devel team mailing list archive
-
meadl-devel team
-
Mailing list archive
-
Message #00080
Re: Accessible options menu
Hi Jorge,
I have pasted my findings on this link-
https://bugs.launchpad.net/meadl/+bug/617060 .
Here are my findings about emulating a physical Keyboard to access the
menu-
-
Earlier we were able to open the options menu using key-event and
currentInputConnection, but it turned out that you cannot access your menu
using those key-events.
-
The input connection accessible to application always points to the
current focused activity and not the menu or pop-ups by that that activity.
So we tried some other available key-injecting methods but none of them was
good enough due to a security exception in android framework.
-
Then, as Jorge suggested, the phsyical keyboard on an android device is
able to navigate the options menu or any other pop-up, and if we could
analyze its behavior, maybe we will also be able to generate the same code.
-
So I began with my testing to analyze what are the flow of events that
happen when an hardware key is pressed. The IME can intercept physical
keyboard events through the onKeyDown() and onKeyUp() methods, as well as
either consume them or process and pass them along to the OS.
-
The onKeyUp and onKeyDown functions receives the Keyevent and key code as
input to the function, and the same key-event and code are passed by these
functions to their respective super methods(super.onKeyUp etc.) as it is, so
that they are processed by OS to trigger the event.
-
If we are able to generate these both inputs to function exactly same
programatically, then we may be able to pass it to these functions from
inside the code to trigger the same event.
-
The keyCode field in the function is an object of type int, and is thus
not difficult to generate. But the other input is of type KeyEvent, and I
did not knew how to re-create an object of this type until I knew what was
contained in this.
-
So I tried to capture the KeyEvent generated by the hardware key and Log
it in string format so that I could see and understand what it contained.
-
I tried searching on net, but there was no solution to the problem of
generating your Hardware like key-event, instead every link just explained
about how to capture a hardware key event. So I started on my own to find a
way and after little bit of trying and experimentation, I was able to Log
the KeyEvent generated by the Hardware key in string format.
-
It looked like this- *KeyEvent{action=0 code=82 repeat=0 meta=0
scancode=108 mFlags=8}*.
-
So the only task left in hand was to generate a similar Key Event
programatically and to pass it to onKeyUp/Down function along with the code.
I started with first understanding what these various fields(meta, repeat
etc.) meant in the result of Key Event format.
-
I found out that only thing we need to worry about generating were the
scancode and mFlags, rest can be generated easily.
-
The mflags=8 here meant that this event was generated by a trusted part
of system and is not spoofed by any 3rd party application. So our task was
to spoof this mFlag to be 8 programatically.
-
I tried various KeyEvent constructors to try to generate the exact same
event. Fortunately, I found a constructor which will allow us to set all the
above values, but also required some other fields to be filled, such as
device id, event down/up time.
-
So I filled the event down/Up time as current time using System clock and
filled device id as 0.
-
After constructing this key-event, I passed it to first the onKeyDown
function and then to onKeyUp function along with the key-code to test if we
could open the menu using this by emulating the menu key event.
-
I printed the key event recieved by both those functions sent by me, and
their value in log was exact same as generated by a hardware key, but menu
did not opened. I tried various other combinations of fields, but even
though both my and the hardware key event showed the same key-event value on
the Log, my version was not able to trigger the menu.
I would continue experimenting to see if there is something I missed or if
something more need to be sent to function. I have also updated the wiki
page - http://wiki.scyp.idrc.ocad.ca/w/Tekla/Accessible_Options_Menu.
Rishabh
On Sat, Apr 16, 2011 at 9:15 PM, Jorge Silva <jsilva@xxxxxxx> wrote:
> Hola Rishabh,
>
> This is great... very detailed and useful. Could I trouble you to
> copy-paste your findings to the following bug report:
> https://bugs.launchpad.net/meadl/+bug/617060 ?
>
> This confirms the limitations we have been facing before. Following on
> your test with the getCurrentInputConnection() method though, I would like
> to do one more test, but we would need a handset with a physical keyboard.
> Does your handset have one? The reason I ask is because the DPad on the
> physical keyboard of my Motorola Droid actually does the right thing (i.e.,
> can be used to navigate the options menu) and I was wondering if capturing
> the connection before and after the menu is open would provide a clue as to
> what the physical keyboard is doing differently and perhaps even how to fix
> the issue.
>
> Another piece of this puzzle is the fact that the IME can intercept
> physical keyboard events through the onKeyDown() and onKeyUp() methods, as
> well as either consume them or process and pass them along to the OS. This
> suggests that we could potentially use those methods to trigger the exact
> same code that gets executed when the physical keyboard is activated if we
> override them with their super.[method] equivalents and call them
> programatically (I just thought of this).
>
> Let me know whether you have access to a handset with a physical keyboard,
> otherwise I can try myself.
>
> cheers!
> Jorge
>
>
> On 11-04-16 09:03 AM, Rishabh Garg wrote:
>
>> Hi Jorge,
>>
>> For some reason, I am not able to push my branch on to launchpad, there
>> is some error coming in bazaar. I would try to fix it and upload my
>> branch as soon as possible.
>> Meanwhile, I have started working on the next problem of giving access
>> the options menu. I also ran the tests you mentioned. Here are my
>> findings from day 2-
>>
>> *
>>
>> I started with adding the code to trigger a sequence of key-events
>> in response to pressing the 'fake options menu key'.
>>
>> *
>>
>> I modified the code for keyDownUp function such that, whenever it
>> received options menu key-code, it would first send the key-event
>> for options menu. Then it waits for 3 seconds before firing in
>> another key-event of D-pad up key. Finally, after another 3 second
>> wait, it would fire the event for D-pad right key.
>>
>> *
>>
>> After trying this code on some applications, it seemed that the
>> Dpad key events were sent to the activity that opened the menu,
>> instead of menu itself.
>>
>> *
>>
>> This may be because the getCurrentInputConnection gives the input
>> connection for the focused activity(as mentioned in its doc), and
>> since menu is not a different activity, but a part of the activity
>> that opened it, so the key-events are sent to the activity that
>> opened it.
>>
>> *
>>
>> I printed the human-readable format of current Input Connection
>> before and after opening the menu on Log, and as expected,both
>> times the input connections had the same value in Log.
>>
>> *
>>
>> This confirms that input connection which is used to send activity
>> does not change even after opening the menu and thus key-events
>> are still sent to activity.
>>
>> *
>>
>> I then moved on to try some other methods to inject key-events.
>> First method I came across was using the IEWindow Manager and the
>> Service Manager.
>>
>> *
>>
>> After doing some research about IEWindow Manager class, I found
>> out that this was available in very early versions of andorid APIs
>> but has been removed since from android public api as its
>> development was not stable and was not safe for future.
>>
>> *
>>
>> Next, I came across the Instrumentation class of android. The main
>> purpose of this class is for unit testing the applications, by
>> generating automated sequence of key-events, and is not
>> recommended for use in normal applications.
>>
>> *
>>
>> The instrumentation class cannot be run in main thread, so I put
>> the key inject function of instrumentation class in a parallel
>> thread. When I tried running the code, pressing on fake menu
>> button generated a security Exception.
>>
>> *
>>
>> The exception stated that Injecting to another application
>> requires inject_event permission.I then started to find more about
>> this error and permission and came across following.
>>
>> *
>>
>> The android.permission.INJECT_EVENT permission is not that of the
>> application, the *WindowManagerService* must have this permission
>> otherwise requests to inject events into other applications'
>> windows will be rejected.
>>
>> *
>>
>> By default, WindowManagerService does not have this permission,
>> that is why it is impossible to generate events for somebody
>> else's window.
>>
>> *
>>
>> The IEWindow manager would also have faced the same problem, if it
>> was to be used in worst case scenario.
>>
>> *
>>
>> There were some official android blogs also that said that it is
>> not possible to inject key-events into another application for
>> security reasons and there is no way around this.
>>
>> http://www.mail-archive.com/android-developers@xxxxxxxxxxxxxxxx/msg13424.html
>>
>> I would continue with my experimentation with existing methods to see if
>> there is still a way around or if there is a new method available.
>> I will also update the wiki with details of each of the above methods
>> and problems in them
>>
>>
>> Rishabh
>>
>>
>> On Fri, Apr 15, 2011 at 8:26 PM, Jorge Silva <jsilva@xxxxxxx
>> <mailto:jsilva@xxxxxxx>> wrote:
>>
>> great job Rishabh, can you please upload your modified branch back
>> to Launchpad so I can check it out? If you have done so already,
>> please just send me the link.
>>
>> BTW. Now that this is solved, I'd like to do a couple more tests
>> before we move on to extracting the contents of the options menu
>> (because we may be able to solve it without having to go that far).
>> Would it be possible for you to add code to trigger a sequence of
>> key events in response to pressing the 'fake' options menu key you
>> have created? Something like this:
>>
>> 1. Press the *modified key* to open the options menu
>> 2. Wait a few seconds (to allow for the options menu to be shown)
>> and then send a DPAD event.
>> 3. Wait a few more seconds and send another DPAD event
>>
>> The goal would be to see if those events get sent to the options
>> menu. If they aren't, then we should try the other methods for
>> injecting key events on the URLs I sent earlier and see if any of
>> those would do the right thing.
>>
>> I believe you can emulate a key press at any time from the IME by
>> just calling the onKey() method with the right code, and to test the
>> other methods of injecting keys, you would just have to modify the
>> keyDownUp() method.
>>
>> Let me know if you have any questions...
>>
>> cheers!
>> Jorge
>>
>>
>> On 11-04-15 08:08 AM, Rishabh Garg wrote:
>>
>> Hi Jorge,
>>
>> I think I have found the solution for triggering the options
>> menu via a
>> key event. I have also tracked why something else was being
>> triggered by
>> the options menu key-code, the problem which Kevin was facing on
>> his
>> device. Here are my findings-
>>
>> *
>>
>> I started by changing the key-code value for one of the
>> navigation
>> key(the right key) to the key-code of the menu key(82) in
>> the xml
>> file. As Kevin mentioned earlier, this was triggering
>> something
>> weird instead of the menu
>>
>> *
>>
>> I then tried replacing the menu key-code by all other
>> possible
>> key-codes to check which of the key-codes behaved
>> differently.
>>
>> *
>>
>> When experimenting with different key-codes resulted in
>> nothing,I
>> decided to backtrack the flow of code that resulted in the
>> key-event.
>>
>> *
>>
>> I printed the key-code received by keyUpDown function in
>> the log,
>> to check if the correct event was received by the function.
>>
>> *
>>
>> After checking the log for various key-codes in xml, I
>> found that
>> the keyUpDown function received key-codes for some
>> specific keys
>> only, and this function was not called for rest of the cases.
>>
>> *
>>
>> I started to find the functions that called the keyUpDown
>> function
>> and analyze them.
>>
>> *
>>
>> I found that that keyUpDown function was called by a handler
>> function for nav bar, which in turn was called by onKey
>> function.
>> For some reason the onKey function was not able to call the
>> handler function for nav bar, and instead called some other
>> function which generated some other event.
>>
>> *
>>
>> It finally turned out that the onKey function uses another
>> function to check key-code it reveives, isNavigationKey
>> function.
>> *
>>
>> isNavigationKey function checks whether the recieved key code
>> belonged to a set of key-codes(up,down,left,right,center
>> or back).
>> If it does, it passes it to KeyUpDown function via handler
>> , but
>> for any other key code, it passes it to
>> handleCharacterfunction,
>> which uses prediction etc. to trigger some other
>> key-event. This
>> is the reason some other event was triggered in Kevin's
>> device.
>> *
>>
>> I added the key-code for menu in the set of key-codes in the
>> isNavigationKey function, which enabled it to pass the
>> key-code
>> for menu to keyUpDown function.
>> *
>>
>> The KeyUpDown function was able to trigger the key-event
>> for menu
>> correctly .
>>
>> I changed the key-code of*right button in nav bar* to act as
>> *menu button*.
>> After checking the new code on emulator, I have also thoroughly
>> checked
>> it on my device. The menu is correctly opened for android
>> homescreen as
>> well as all 3rd party applications(Gmail,Gtalk,
>> Youtube etc.) by the right button in nav bar.
>>
>> I have uploaded the new apk with these change on this link-
>> http://web.iiit.ac.in/~rishabh.garg/IMESettings.apk
>> <http://web.iiit.ac.in/%7Erishabh.garg/IMESettings.apk>
>> <http://web.iiit.ac.in/%7Erishabh.garg/IMESettings.apk>. Could you
>>
>> please confirm if the above solution is working fine or if there
>> is any
>> problem in it.
>> I would also update my findings on this wiki page -
>> http://wiki.scyp.atrc.utoronto.ca/w/Tekla/Accessible_Options_Menu.
>> .I think this solves the first 2 points of our solution blueprint-
>> 1. Open the options menu programatically
>> 2. Detect the event triggered when the options menu is open,
>> Regards
>> Rishabh Garg
>>
>>
>> On Thu, Apr 14, 2011 at 3:09 AM, Jorge Silva
>> <jsilva.idi@xxxxxxxxx <mailto:jsilva.idi@xxxxxxxxx>
>> <mailto:jsilva.idi@xxxxxxxxx <mailto:jsilva.idi@xxxxxxxxx>>>
>> wrote:
>>
>> Hola,
>>
>> As many of you already know, Tekla users are still locked
>> out the
>> options menu because of two reasons:
>> 1. When the menu pops-up, it occludes the keyboard, and
>> 2. Even when the keyboard remains active, its key events are
>> sent to the
>> activity underneath, instead of to the menu on top
>>
>> The Google accessibility team has confirmed the issue is in
>> the Android
>> framework and they are working on it, but we'll try a
>> workaround that
>> has been drafted in the following blueprint:
>>
>> https://blueprints.launchpad.net/meadl/+spec/accessible-options-menu
>>
>> Long-story short, in order to make this happen, the first
>> thing we need
>> to do is make sure Tekla can open the options menu on any app,
>> programatically. The Goldeneye
>>
>> (http://wiki.scyp.atrc.utoronto.ca/w/Goldeneye_Reading_Project)
>> project
>> devs already tried to do so by sending the corresponding key
>> event but
>> the device they tested on wasn't very cooperative. We need
>> to do the
>> following:
>> 1. Confirm whether in fact it is not possible to send the
>> key event
>> using the current methods (code is at: http://goo.gl/dOenl)
>> 2. If the key event doesn't work, try other methods like:
>> http://stackoverflow.com/questions/3133318/open-the-options-menu-
>> programatically or
>>
>> http://mylifewithandroid.blogspot.com/2009/01/generating-keypresses-
>> programmatically.html
>> <
>> http://mylifewithandroid.blogspot.com/2009/01/generating-keypresses-%0Aprogrammatically.html
>> >
>>
>> 3. If none of that works, then we should try looking for an
>> intent
>>
>> Let me know if any of you would like to help trying this
>> out... I am a
>> bit tied at the moment with other commitments. If you have
>> any other
>> questions please feel free to reply to the thread.
>>
>> cheers!
>> Jorge
>> --
>> This message was sent from Launchpad by
>> Jorge Silva (https://launchpad.net/~jorge-silva
>> <https://launchpad.net/%7Ejorge-silva>
>> <https://launchpad.net/%7Ejorge-silva>)
>>
>> using the "Contact this team" link on the Tekla development
>> team
>> team page
>> (https://launchpad.net/~meadl-devel
>> <https://launchpad.net/%7Emeadl-devel>
>> <https://launchpad.net/%7Emeadl-devel>).
>>
>> For more information see
>> https://help.launchpad.net/YourAccount/ContactingPeople
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~meadl-devel
>> <https://launchpad.net/%7Emeadl-devel>
>> <https://launchpad.net/%7Emeadl-devel>
>>
>> Post to : meadl-devel@xxxxxxxxxxxxxxxxxxx
>> <mailto:meadl-devel@xxxxxxxxxxxxxxxxxxx>
>> <mailto:meadl-devel@xxxxxxxxxxxxxxxxxxx
>> <mailto:meadl-devel@xxxxxxxxxxxxxxxxxxx>>
>>
>> Unsubscribe : https://launchpad.net/~meadl-devel
>> <https://launchpad.net/%7Emeadl-devel>
>> <https://launchpad.net/%7Emeadl-devel>
>>
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>> --
>> Rishabh Garg
>> B.Tech CSE 2nd year
>> International Institute of Information Technology(IIIT)
>> Hyderabad,India
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~meadl-devel
>> <https://launchpad.net/%7Emeadl-devel>
>> Post to : meadl-devel@xxxxxxxxxxxxxxxxxxx
>> <mailto:meadl-devel@xxxxxxxxxxxxxxxxxxx>
>> Unsubscribe : https://launchpad.net/~meadl-devel
>> <https://launchpad.net/%7Emeadl-devel>
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~meadl-devel
>> <https://launchpad.net/%7Emeadl-devel>
>> Post to : meadl-devel@xxxxxxxxxxxxxxxxxxx
>> <mailto:meadl-devel@xxxxxxxxxxxxxxxxxxx>
>> Unsubscribe : https://launchpad.net/~meadl-devel
>> <https://launchpad.net/%7Emeadl-devel>
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>> --
>> Rishabh Garg
>> B.Tech CSE 2nd year
>> International Institute of Information Technology(IIIT)
>> Hyderabad,India
>>
>>
--
Rishabh Garg
B.Tech CSE 2nd year
International Institute of Information Technology(IIIT)
Hyderabad,India
Follow ups
References
