sikuli-driver team mailing list archive

[RaiMan <question698511@xxxxxxxxxxxxxxxxxxxxx>]

Question #698511 on SikuliX changed:
https://answers.launchpad.net/sikuli/+question/698511

Description changed to:
The issue will be tracked on GitHub: https://github.com/RaiMan/SikuliX1/issues/474
------------------------------------------------------------------

Java Version: 11
Sikuli API Version: 2.0.5
Computers Tested: 13" M1 2020 MacBook Pro and 13" 2017 MacBook Pro
OS Version: macOS Big Sur 11.4

I have a MacbookPro with a Retina screen connected to an external
display. The external display has a resolution of 2560x1440 but is not
treated as Retina by MacOS (Big Sur). I have the external display
configured to be the primary via macOS Display settings, making it
Screen 0 for Sikuli . Sikuli sees Screen 1 as the native screen of the
MacBook Pro (Retina) positioned to the left through MacOS Display
configuration. Here is Screen.showMonitors() from Java using Sikuli API:

*** monitor configuration [ 2 Screen(s)] ***
*** Primary is Screen 0
Screen 0: R[0,0 2560x1440]@S(0)
Screen 1: R[-1680,284 1680x1050]@S(1)
*** end monitor configuration ***

With the configuration this way, I can reliably capture/find images on
Screen 0, but the pixel dimensions are so far off on Screen 1 (the
Retina screen) captures and finds fail. Sikuli API doesn't seem to be
capturing things correctly on the Retina screen in this configuration.

See https://github.com/harmonjeff/sikuli-retina-
bug/tree/master/FailedCaptureImages

To illustrate the issue I created a test Java application that uses the
Sikuli API to capture the screens. I opened up two windows of TextEdit,
placing one on each screen. I sized the TextEdit window to cover the
entire screen (except the dock on Screen 0). You can see the source
here:

https://github.com/harmonjeff/sikuli-retina-bug

If I switch the primary window to being the Retina screen using the
MacOS Display settings and re-run the test then the Retina screen
captures correctly and the external display does not.  It seems the mix
of Retina with non-Retina causes the non-primary screen to not
capture/find correctly.

In researching what could be going on I discovered a system property
that seems to make things function correctly as far as Siklui API
capturing and finding images using Java. If I set the
"sun.java2d.uiScale" property to be "1.0" then the captures in the
sample application function correctly on both screens.

https://github.com/harmonjeff/sikuli-retina-
bug/tree/master/SuccessCaptureImages

Setting that system parameter in the launch of the JVM or in a static
that executes before any Java code tries to use AWT seems to fix Siklui
API capture/find methods on both screens.  This solution works for me
with one significant exception.  I need to be able to tell if the
application I am testing using Sikuli is running on a Retina screen.
Images captured using Siklui on the Retina display do not match well
when the application is on the non-Retina screen and vice-versa. I am
solving this problem by capturing reference images from both Retina and
non-Retina, but I need to know if the application is running on a Retina
screen to use the correct reference images.

The best way I have seen to detect if an application window is on a
Retina screen in Java 9+ is in the same application code shared above,
the isRetina method. The method only works if the system property is not
defined. It seems that if you use the system property to set the UI
scaling to 1.0 then it applies to all screens and the AffineTransform no
longer has an accurate result on the isIdentity method.  I tried
detecting the Retina screens just before setting that system property,
but it seems that as soon as I access the AWT objects to do that test
setting the system property doesn't do anything.

I don't have a way to both detect the application is running in a Retina
screen and have the capture/find Sikuli API functions work properly on
both screens when one is Retina and the other is not.

This issue seems very closely related to this question: https://answers.launchpad.net/sikuli/+question/682295 
Which resulted in the creation of this bug: https://bugs.launchpad.net/bugs/1840689

-- 
You received this question notification because your team Sikuli Drivers
is an answer contact for SikuliX.