sikuli-driver team mailing list archive

[Jeff Harmon <question698511@xxxxxxxxxxxxxxxxxxxxx>]

New question #698511 on SikuliX:
https://answers.launchpad.net/sikuli/+question/698511

Java Version: 11
Sikuli API Version: 2.0.5
Computers Tested: 13" M1 2020 MacBook Pro and 13" 2017 MacBook Pro
OS Version: macOS Big Sur 11.4

I have a MacbookPro with a Retina screen connected to an external display. The external display has a resolution of 2560x1440 but is not treated as Retina by MacOS (Big Sur). I have the external display configured to be the primary via macOS Display settings, making it Screen 0 for Sikuli . Sikuli sees Screen 1 as the native screen of the MacBook Pro (Retina) positioned to the left through MacOS Display configuration. Here is Screen.showMonitors() from Java using Sikuli API:

*** monitor configuration [ 2 Screen(s)] ***
*** Primary is Screen 0
Screen 0: R[0,0 2560x1440]@S(0)
Screen 1: R[-1680,284 1680x1050]@S(1)
*** end monitor configuration ***

With the configuration this way, I can reliably capture/find images on Screen 0, but the pixel dimensions are so far off on Screen 1 (the Retina screen) captures and finds fail. Sikuli API doesn't seem to be capturing things correctly on the Retina screen in this configuration.

See https://github.com/harmonjeff/sikuli-retina-bug/tree/master/FailedCaptureImages

To illustrate the issue I created a test Java application that uses the Sikuli API to capture the screens. I opened up two windows of TextEdit, placing one on each screen. I sized the TextEdit window to cover the entire screen (except the dock on Screen 0). You can see the source here:

https://github.com/harmonjeff/sikuli-retina-bug

If I switch the primary window to being the Retina screen using the MacOS Display settings and re-run the test then the Retina screen captures correctly and the external display does not.  It seems the mix of Retina with non-Retina causes the non-primary screen to not capture/find correctly.

In researching what could be going on I discovered a system property that seems to make things function correctly as far as Siklui API capturing and finding images using Java. If I set the "sun.java2d.uiScale" property to be "1.0" then the captures in the sample application function correctly on both screens.

https://github.com/harmonjeff/sikuli-retina-bug/tree/master/SuccessCaptureImages

Setting that system parameter in the launch of the JVM or in a static that executes before any Java code tries to use AWT seems to fix Siklui API capture/find methods on both screens.  This solution works for me with one significant exception.  I need to be able to tell if the application I am testing using Sikuli is running on a Retina screen.  Images captured using Siklui on the Retina display do not match well when the application is on the non-Retina screen and vice-versa. I am solving this problem by capturing reference images from both Retina and non-Retina, but I need to know if the application is running on a Retina screen to use the correct reference images.

The best way I have seen to detect if an application window is on a Retina screen in Java 9+ is in the same application code shared above, the isRetina method. The method only works if the system property is not defined. It seems that if you use the system property to set the UI scaling to 1.0 then it applies to all screens and the AffineTransform no longer has an accurate result on the isIdentity method.  I tried detecting the Retina screens just before setting that system property, but it seems that as soon as I access the AWT objects to do that test setting the system property doesn't do anything.

I don't have a way to both detect the application is running in a Retina screen and have the capture/find Sikuli API functions work properly on both screens when one is Retina and the other is not.

This issue seems very closely related to this question: https://answers.launchpad.net/sikuli/+question/682295 
Which resulted in the creation of this bug: https://bugs.launchpad.net/bugs/1840689

-- 
You received this question notification because your team Sikuli Drivers
is an answer contact for SikuliX.