sikuli-driver team mailing list archive

[RaiMan <question667037@xxxxxxxxxxxxxxxxxxxxx>]

Question #667037 on Sikuli changed:
https://answers.launchpad.net/sikuli/+question/667037

    Status: Open => Answered

RaiMan proposed the following answer:
Though I think, that to use Python language is not too complicated and
challenging to write down basic to medium complex SikuliX based
workflows, there is always room, for making it even easier for people,
that should not be "tortured with programming".

With SikuliX workflows we have 3 challenges, to support these people:

1. a frontend, that easily allows to add the SikuliX features to a
workflow, based on clickable templates, that "know" about the already
existing workflow context and support the intuitive completion of specs
and parameters. This should be available for automation workflows and
testcases. Running such a workflow means to translate it to a runnable
script and run it.

2. a tool, that allows to capture, optimize and manage the images needed
in a workflow.

3. a recorder, that allows to setup a basic workflow while working with
some application.

I am on the way to address all 3 aspects with SikuliX2, but have to
admit, that I am some months away from a useable package.

my ideas about the 3 challenges (for all 3 there are already basic
approaches available in SikuliX2):

-- at 1: the frontend should be table-based (like the workflows with RobotFramework).
Each line is an action like click, wait, type/write/paste.
An action can be selected from a context-menu or entered by typing in the first column (auto-complete) and a suitable template is filled into the line.
Parameters of an action are edited by just clicking on the respective column in the line (example: you can capture and/or manage a needed image supported by the tool of challenge 2)
With this basic sequential workflows can be created and run easily.
Medium complex workflows additionally need repeat (loop, for, ..) and decision (if/elif/else) features, some kind of supporting functions and import and some features to interact with the user, while the workflow is executed.
In SikuliX2 I already have a basic approach based on a tablebased JPanel, that translates into a runnable JavaScript script (JavaScript with Java has the advantage, that the interpreter is JVM-builtin (Nashorn))
After having implemented such a frontend, it might be easily complemented with a completely visual tool, that shows the workflow as a diagram with graphical elements, that can be edited by just clicking and dragging around (you mentioned UML).
It might be a good idea, to make this frontend browser-based.
And of course it could be implemented in Eclipse/Netbeans/IntelliJ IDEA using their plugin frameworks or even in JEdit.

-- at 2: the tool must allow to capture images and optimize them for the use of SikuliX features (concentrate on the key visual aspects, avoiding background towards the edges, allow transparency and images in parts and last but not least manage image sets).
The challenge here is a suitable GUI. This tool can then be used as a library in aspects 1 and 3.

-- at 3: this is the most challenging, since the normal approaches with Java are not useable (to catch mouse and keyboard actions you always need some overlay frames, which are not possible in all cases (e.g. full screen) and all over the place).
The most promising approach is using the JNativeHook library, that allows to run a recorder in the background that tracks all mouse and keyboard actions and decides wether it is a command for the recorder or going through to the running application. In the end you have some raw workflow, that can be used as is or optimised/augmented with the tool 1. The recorder also tracks screen content using intermediate screenshots in various aspects depending on the state of recorder and/or application. Images might be edited while recording or afterwards based on the screenshots using tool 2.

Conclusion: I think your ideas go into the right direction and I am very
interested, to further discuss the requirements with you.

Maybe we can agree on some toolbox, that might be used in your Eclipse
based approach  and in SikuliX2 as well.

You received this question notification because your team Sikuli Drivers
is an answer contact for Sikuli.