sikuli-driver team mailing list archive

[Raymond Gonzales <question106059@xxxxxxxxxxxxxxxxxxxxx>]

Question #106059 on Sikuli changed:
https://answers.launchpad.net/sikuli/+question/106059

Raymond Gonzales proposed the following answer:
Hello!

I have a way to get text from any screen using sikuli and another tool
called Accessibility Inspector.  Accessibility Inspector is tool that
comes with Xcode 4.  You would need to get Xcode 4 here -
http://developer.apple.com/xcode/

Once you have Xcode 4 fully installed, you can open the app manually by
typing the following search query in spotlight - Accessibility
Inspector.


Now for the fun....

I am programming in java so bear with me....

SAMPLE CODE

and just so you know before you go reading...

I use 's' as my sikuli object so anything you see that starts with s.
such as s.click is a sikuli command

I also create a sikuli object called 'app'

Really...now time for the fun...

I first open the app using the following command:

            app.openApp("Accessibility Inspector");

-----------------------------------------------------------------------------

you may have to position the Accessibility Inspector app in a place that
doesnt interfere with the image you are trying to get text from.....

-----------------------------------------------------------------------------

Now I pause the java thread to allow the Accessibility Inspector app to
load then I do a "find" for the image I want

BTW MobileUtils is a class I created that has helper methods that I
created for myself...

                MobileUtils.thinktime(10000);

                s.find([IMAGE I AM LOOKING FOR]);

-----------------------------------------------------------------------------

The find command allows me to get the coordinates of the image I am
looking for. I get an x coordinate by doing the following command:

                int xCoordinate = s.getLastMatch().getX() + 10;

-----------------------------------------------------------------------------

I get an y coordinate by doing the following command:

                int yCoordinate = s.getLastMatch().getY() + 100;

-----------------------------------------------------------------------------

I move the mouse to the area where the text I am looking for is at by feeding a mouseMove method x and y coordinates
Just in case you dont know about this Java class...here you go - http://www.1your.com/drupal/robotclassdemo

                robot.mouseMove(xCoordinate, yCoordinate);

-----------------------------------------------------------------------------

By this time the Accessibility Inspector app is the current active app so what I do is
I copy all the contents of whats in the Accessibility Inspector app,  which should have 
the text I am looking for, by telling my comp to select Keys control A to select all text, then control c, 
to copy the text, and finally control q to quit the Accessibility Inspector app

BTW the KeyboardMouseCommands is a class I created...it has all my
keyboard and mouse commands that I use... :)

                MobileUtils.thinktime(10000);

                KeyboardMouseCommands.SELECT_KEYS_COMMAND_A();

                KeyboardMouseCommands.SELECT_KEYS_COMMAND_C();

                KeyboardMouseCommands.SELECT_KEYS_COMMAND_Q();

-----------------------------------------------------------------------------

Finally I get the contents I just copied from the Accessibility
Inspector app off of my system clipboard by using a clipboard method I
got off the web -
http://www.exampledepot.com/egs/java.awt.datatransfer/ToClip.html

                String textInCell = MobileUtils.getClipboard();


I automate iphone apps and this works well with simulator iphone apps and all programs built for your mac or iphone or ipad. You cannot use this to automate Android apps in an Android emulator because the Accessibility Inspector app doesnt know the contents of the emulator (makes sense :P)....

-- 
You received this question notification because you are a member of
Sikuli Drivers, which is an answer contact for Sikuli.