sikuli-driver team mailing list archive

Thread
Date
[Question #218011]: Batch to create Sukili XML report

To: sikuli-driver@xxxxxxxxxxxxxxxxxxx
From: aurnoi1 <question218011@xxxxxxxxxxxxxxxxxxxxx>
Date: Mon, 31 Dec 2012 21:50:58 -0000
Reply-to: question218011@xxxxxxxxxxxxxxxxxxxxx
Sender: bounces@xxxxxxxxxxxxx
New question #218011 on Sikuli:
https://answers.launchpad.net/sikuli/+question/218011

Hi everyone!

First I would like to thanks the team in charge of Sikuli. I am using it since one week and it’s a wonderful tool for test automation (especially on “WindowsXpeLimitedRessources” platforms…). I would like to contribute with a little batch I wrote (at the end of this text) which will present the result of the scripts executed by Sikuli in XML file.

I have almost no experience with Sikuli (nor XML terminology), so some unexpected behaviors could appear. Any information returns will be appreciated!

The XML form:
---------------------------------------------------
:: <TestResult>
::    <TestsFailedCount></TestsFailedCount>
::    <StartTime></StartTime>
::    <EndTime></EndTime>
::    <Duration></Duration>
::       <Test>
::          <ScriptName></ScriptName>
::          <ScriptPath></ScriptPath>
::          <ScriptArgs></ScriptArgs>
::          <Log></Log>
::          <ExecDuration></ExecDuration>
::          <ExecDurationInSec></ExecDurationInSec>
::          <TestStatus></TestStatus>
::       </Test>
::       <Test>
::           …
::       </Test>
:: </TestResult>
---------------------------------------------------

Where:

TestResult: The root holding all the tests executed.
TestsFailedCount: The total of failing tests.
StartTime: The starting date/time of the batch.
EndTime: The ending date/time of the batch.
Duration: The total duration since the batch has been executed, displayed in hours:minutes:seconds:ms. 
Test: An object “Test”, where was executed a Sikuli script, with the fowling attributes:
	ScriptName: Name of the script (e.g.: “OpenControlPanel.sikuli”).
	ScriptPath: The complete path to the script (e.g.: “D:\SikuliTests\OpenControlPanel.sikuli”).
	ScriptArgs: The argument(s) passed to the Sikuli script (e.g.: “123 “arg with space”).
Log: The log ouput as seen in IDE when the Sikuli script is run but also the Java errors. This is based on the “error” word which I have observed at runtime (e.g.: The image to click doesn’t exists) and while calling Java (e.g.: When misspelling –args, which is case sensitive).
ExecDuration: The execution duration of the Sikuli script displayed in hours:minutes:seconds:ms.
ExecDurationInSec: The execution duration of the Sikuli script displayed in second.
TestStatus: The test status expressed in Boolean (“TRUE” for “PASS” and “FALSE” for “FAIL”).


Prerequisites:

This batch file needs two helpers (only executables, nothing to install!):
 “fart-it” (http://sourceforge.net/projects/fart-it/”;).
“XMLStartlet” (http://sourceforge.net/projects/xmlstar/?source=navbar). 

The first one helps to replace strings during the process. The second formats the temporary log (“TestResult.txt”) to xml file (“TestResult.xml”).

Tested versions:
 fart199b_win32 and xmlstarlet-1.4.2-win32.


Disclaimer:

I have only tested on Windows XP Pro - 32bits -English version. If you experience problems on Windows Vista/7/8 check if your UAC is disabled. The native language of your OS may have an impact too. 

Also, tags “<>” may appears in the log generated by Sikuli or Java. To avoid problems when formatting the XML, I have replaced these chars: 
“<” = “{”
“>” = “}”.


Syntax:

>From a batch: 
First be sure that “fart.exe” and “xml.exe” are in the same directory than “sikuliTestResult.cmd”.
Then you may create a batch (e.g.: DriveSikuliTests.cmd) from where your calls to Sikuli scripts (with arguments too) will be passed. At the end of this batch you have to call “sikuliTestResult.cmd” with the argument “/C”, to convert the test log to XML file.

The log “TestResult.xml” will be in the same directory than “sikuliTestResult.cmd”.

 EXAMPLE from a batch file:
:: Set the actual path of the batch.
SET actualPath=%~dp0
SET actualPath=%actualPath:~0,-1%

:: Call two Sikuli scripts.
CALL "%actualPath%\sikuliTestResult.cmd" "D:\SikuliScripts\PassTest.sikuli"
CALL "%actualPath%\sikuliTestResult.cmd" "D:\SikuliScripts\ShowArgs.sikuli" --args 123 "arg with spaces."

:: Convert to xml
CALL "%actualPath%\sikuliTestResult.cmd" /C


>From application with shell execution support: 
Also, you may call the batch “sikuliTestResult.cmd” from application with shell execution support. Each time a Sikuli script is executed, “sikuliTestResult.cmd” returns a STDERR to know the result of the test. If appears the word “error” (not case sensitive) in the log of your test (during the script execution or in Java VM), the STRERR will return the sum of the “error” word occurrences. “0” (zero) meaning your test is successful.

At the end of your test session, you still have to call “sikuliTestResult.cmd” with the argument “/C”, to convert the test log to XML file.

TIPS: In your Sikuli scripts, you may print “error” or “[Error]” when an expected result is not realized.

EXAMPLE to get STDERR from command prompt:
CALL "D:\sikuliTestResult.cmd" "D:\SikuliScripts\PassTest.sikuli"
Echo %errorlevel%
0
CALL "D:\sikuliTestResult.cmd" "D:\SikuliScripts\FailTest.sikuli"
Echo %errorlevel%
1


Good Practice:

Name your Sikuli script as you would like to name the test attached. So it will be more readable in the XML.

Before to call “sikuliTestResult.cmd” remove all file which may have been created during a previous session of test using “sikuliTestResult.cmd” (in the same directory that this last one). These files are deleting each time you complete transformation to xml. But if something went wrong during the execution of the batch… C'mon it’s just a batch! What would you seriously expected??? : )

Here is the batch:

*********** COPY THE REST OF THE TEST TO A BATCH NAMED "sikuliTestResult.cmd" *****************

:: SikuliTestResult.cmd v1.0.0.1

:: This batch provides a TestResult.xml, so you can import this xml to EXCEL or any deserializer software.
:: STDOUT and STDERR are redirect to the TestResult.
:: First you have to your tests (their results will be written to "TestResult.txt")
:: and finally call again this batch with /C as argument, to convert "TestResult.txt" to "TestResult.xml".
::
:: *******************************************
:: ************** Dependencies ***************
:: This batch file needs two helpers (only executable, nothing to install!):
:: "fart-it" (http://sourceforge.net/projects/fart-it/”;).
:: "XMLStartlet" (http://sourceforge.net/projects/xmlstar/?source=navbar). 
:: 
:: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
:: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
:: $$$$$$$$$                                                                               $$$$$$$$$ 
:: $$$$$$$$$ COPY "fart.exe" AND "xml.exe" IN THE SAME DIRECTORY THAN SikuliTestResult.cmd $$$$$$$$$
:: $$$$$$$$$                                                                               $$$$$$$$$ 
:: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
:: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
:: 
:: The first one helps to replace strings during the process. 
:: The second formats the temporary log ("TestResult.txt") to xml file ("TestResult.xml").
::
:: Tested version:
:: fart199b_win32 and xmlstarlet-1.4.2-win32.
::
:: *******************************************
:: ***************** Syntax ******************
:: *******************************************
:: -	To execute and log a test: 
:: 					CALL "Path_To\sikuliTestResult.cmd" "Path_To_Script.sikuli"
::				    or 
:: 					CALL "Path_To\sikuliTestResult.cmd" "Path_To_Script.sikuli" --args 123 "arg with space"

::      If the test successed %errorlevel% (STDERR) is 0. Any other numbers is the sum of "error" words displayed in the log.
:: 		you can get the %errorlevel% from the command promt: echo %errorlevel%
::
:: -	To convert the "TestResult.txt" to XML file: 
:: 					sikuliTestResult.cmd /C
::
:: *******************************************
:: ************ THE XML STRUCTURE ************
:: *******************************************
:: <TestResult>
::    <TestsFailedCount></TestsFailedCount>
::    <StartTime></StartTime>
::    <EndTime></EndTime>
::    <Duration></Duration>
::       <Test>
::          <ScriptName></ScriptName>
::          <ScriptPath></ScriptPath>
::          <ScriptArgs></ScriptArgs>
::          <Log></Log>
::          <ExecDuration></ExecDuration>
::          <ExecDurationInSec></ExecDurationInSec>
::          <TestStatus></TestStatus>
::       </Test>
::       <Test>
::         …
::       </Test>
:: </TestResult>


:: Let's ROCK!
SET actualPath=%~dp0
SET actualPath=%actualPath:~0,-1%
SET LogFileName=TestResult.txt
SET logFilePath=%actualPath%\%LogFileName%

:: If the argument is /C or /c (for "ConvertToXML) meaning all SIKULI scripts have been executed and
:: you want to convert the result file (*.txt) as XML.
:: The code under this line won't be executed.
IF /I %1==/C GOTO :ConvertToXML

:: startTRXml = File of start time of TestResult (with XML format). Will be merge to the TestResult.xml to know when the TestResult as started.
SET startTRXml=%actualPath%\startTRXml.txt
:: startTRTime = File of start time of TestResult (with %Time% format). Will be used to calculate the duration of TestResult.cmd execution.
SET startTRTime=%actualPath%\startTRTime.txt
SET actualTime=%Time%
SET exitCode=
IF NOT EXIST "%logFilePath%" (
ECHO.^<StartTime^>%Date% %actualTime%^</StartTime^>>> "%startTRXml%"
ECHO.%actualTime%>"%startTRTime%"
:: Create a blank "TestResult.txt" where all the test results ("tempTestLog.txt") will be merged.
ECHO.>> "%logFilePath%"
)

:: Run the SIKULI script pass as argument (with all his args too) which will generated a file "tempTestLog.txt".
:: This file is used to log temporary the information about the actual test.
CALL :runScript %*
EXIT /B %exitCode%
GOTO:EOF



:: Run the script passed as argument and format the "tempTestLog.txt" with XML tags.
:runScript
SET passArgs=
SET tempTestLog=%actualPath%\tempTestLog.txt
FOR /F "tokens=1 delims=-" %%A IN ("%*") DO (SET scriptPath=%%~dpnxA)
FOR /F "tokens=1 delims=-" %%A IN ("%*") DO (SET actualScript=%%~nxA)
SET actualScript=%actualScript%
:: Set passArgs (if there are arguments).
:: Note that "args" (of --args) is seen as an argument too. Reason why passArgs=--%passArgs%.
FOR /F "tokens=2 delims=-" %%A IN ("%*") DO (SET passArgs=%%A)
SET firstLetterArgs=%passArgs:~0,1%
IF NOT "%firstLetterArgs%"=="~0,1" (
SET passArgs=--%passArgs%
) ELSE (
SET passArgs=
)

:: Write some information in the "tempTestLog.txt".
ECHO.^<Test^>>> "%tempTestLog%"
ECHO.^<ScriptName^>%actualScript%^</ScriptName^>>> "%tempTestLog%" 
ECHO.^<ScriptPath^>%scriptPath%^</ScriptPath^>>> "%tempTestLog%" 
ECHO.^<ScriptArgs^>%passArgs%^</ScriptArgs^>>> "%tempTestLog%" 
ECHO.^<Log^>>> "%tempTestLog%" 2<&1

:: Set the start execution time.
SET startTime=%Time%

:: **** This line will run the script passed as argument (with his arguments too). ****
java -jar -Dsikuli.Debug=0 "%SIKULI_HOME%\sikuli-ide.jar" -s -r "%scriptPath%" %passArgs% >> "%tempTestLog%" 2<&1

:: Close the Log path since the Sikuli script has been executed.
ECHO.^</Log^>>> "%tempTestLog%" 2<&1

:: Log the execution time in format "hours:minutes:seconds:ms" and the total in second.
SET endTime=%Time%
SET timeExec=
SET timeExecS=
CALL :GetTimeExecution %startTime% %endTime%
ECHO.^<ExecDuration^>%timeExec%^</ExecDuration^>>> "%tempTestLog%"
ECHO.^<ExecDurationInSec^>%timeExecS%^</ExecDurationInSec^>>> "%tempTestLog%"

:: Write the <TestStatus> and close the path </Test>.
CALL :GetTestStatus
ECHO.^<TestStatus^>%testStatus%^</TestStatus^>>> "%tempTestLog%"
ECHO.^</Test^>>> "%tempTestLog%"

:: Merge "tempTestLog.txt" and "TestResult.txt".
COPY "%logFilePath%"+"%tempTestLog%" "%logFilePath%" /Y /B
DEL /Q "%actualPath%\TempTestStatus.txt"
DEL /Q "%tempTestLog%"
GOTO:EOF



:: Get the execution duration.
:: Thanks to Scott Stafford and Luke Sampson.
:: http://stackoverflow.com/questions/673523/how-to-measure-execution-time-of-command-in-windows-command-line.
:GetTimeExecution
SET startTime=%1
SET endTime=%2
SET options="tokens=1-4 delims=:."
FOR /f %options% %%a IN ("%startTime%") DO SET startTime_h=%%a&SET /A startTime_m=100%%b %% 100&SET /A startTime_s=100%%c %% 100&SET /A startTime_ms=100%%d %% 100
FOR /f %options% %%a IN ("%endTime%") DO SET endTime_h=%%a&SET /A endTime_m=100%%b %% 100&SET /A endTime_s=100%%c %% 100&SET /A endTime_ms=100%%d %% 100
SET /A hours=%endTime_h%-%startTime_h%
SET /A mins=%endTime_m%-%startTime_m%
SET /A secs=%endTime_s%-%startTime_s%
SET /A ms=%endTime_ms%-%startTime_ms%
IF %hours% lss 0 SET /A hours = 24%hours%
IF %mins% lss 0 SET /A hours = %hours% - 1 & SET /A mins = 60%mins%
IF %secs% lss 0 SET /A mins = %mins% - 1 & SET /A secs = 60%secs%
IF %ms% lss 0 SET /A secs = %secs% - 1 & SET /A ms = 100%ms%
IF 1%ms% lss 100 SET ms=0%ms%
SET /A totalsecs = %hours%*3600 + %mins%*60 + %secs% 
SET timeExec=%hours%h:%mins%m:%secs%s.%ms%ms
SET timeExecS=%totalsecs%.%ms%
GOTO:EOF



:: Verify if there is an "error" word in the log and set the property "<TestStatus>" to $PASS$ or $FAIL$ (will be changed to "True" or "False" when converted to XML).
:: It also set the exitCode ("STDERR" / "%errorlevel%") of this session.
:GetTestStatus
SET testStatus=
TYPE "%tempTestLog%" | FINDSTR /i "error" "%tempTestLog%" | FIND /C /I "error" >> "%actualPath%\TempTestStatus.txt"
SET numOfError=0
FOR /F "usebackq" %%Y IN ("%actualPath%\TempTestStatus.txt") DO (SET /A numOfError=%%Y+%numOfError%)
SET testStatus=$PASS$
IF NOT "%numOfError%"=="0" (
SET testStatus=$FAIL$
SET /A exitCode=%numOfError%
)
GOTO:EOF



:: Convert the "TestResult.txt" to XML.
:: Add the TestFailedCount, StartTime, EndTime and duration of TestResult execution.
:ConvertToXML
IF EXIST "%actualPath%\LogResult.xml" (DEL /Q "%actualPath%\LogResult.xml")

:: Insert the <TestResult>.
SET TRTempFile=%actualPath%\TRTempFile.txt
ECHO.^<TestResult^>> "%TRTempFile%"

:: Log the total of tests which have failed to "TRTempFile.txt".
CALL :GetNumOfTestFail
ECHO.^<TestsFailedCount^>%exitCode%^</TestsFailedCount^>>> "%TRTempFile%"

:: Merge and "startTRXml.txt" with "TRTempFile.txt".
type "%startTRXml%" >> "%TRTempFile%"

:: Set and log the end time of the TestResult execution.
:: Log the duration of the TestResult execution in "TRTempFile.txt".
FOR /F "usebackq" %%T IN ("%startTRTime%") DO SET testResultStartTime=%%T
DEL /Q "%startTRTime%"
SET testResultEndTime=%Time%
SET timeExec=
CALL :GetTimeExecution %testResultStartTime% %testResultEndTime%
ECHO.^<EndTime^>%Date% %testResultEndTime%^</EndTime^>>> "%TRTempFile%"
ECHO.^<Duration^>%timeExec%^</Duration^>>> "%TRTempFile%"

:: Merge "TestResult.txt" with "TRTempFile.txt" and rename this last one as "TestResult.txt".
type "%logFilePath%" >> "%TRTempFile%"
type "%TRTempFile%">"%logFilePath%"
DEL /Q "%TRTempFile%"
DEL /Q "%startTRXml%"

:: Close the TestResult root path.
ECHO.^</TestResult^>>> "%logFilePath%"

:: Finish to format "TestResult.txt" as xml. Now you should have a "TestResult.xml"!
CALL :FinishFormatXML
RENAME "%logFilePath%" "LogResult.xml"
GOTO:EOF



:: Get the number of test which has failed and write it to "TestResult.txt".
:: Set the exitCode (%errorlevel%) to this number of failure.
:GetNumOfTestFail
TYPE "%logFilePath%" | FINDSTR /i /n /l "$FAIL$" "%logFilePath%" | FIND /C "$FAIL$" > "%actualPath%\TempExitCode.txt"
FOR /F "usebackq" %%A IN ("%actualPath%\TempExitCode.txt") DO SET exitCode=%%A
DEL /Q "%actualPath%\TempExitCode.txt"
REM ECHO.^<TestsFailedCount^>%exitCode%^</TestsFailedCount^>>> "%logFilePath%"
:: Change $FAIL$ for FALSE and $PASS$ TRUE in "TestResult.txt".
TYPE "%logFilePath%"  | "%actualPath%\fart.exe" -r -i "%logFilePath%" $FAIL$ FALSE
TYPE "%logFilePath%"  | "%actualPath%\fart.exe" -r -i "%logFilePath%" $PASS$ TRUE
GOTO:EOF



:: Finish to format the log as XML.
:: Since "<" and ">" signs may appears in the log and corrupt the xml,
:: replace "<" and ">" signs by "{" and "}".
:: But first modify the valid tags so they won't be affected.
:FinishFormatXML
CALL :ReplaceXMLTags TestResult
CALL :ReplaceXMLTags Test
CALL :ReplaceXMLTags ScriptName
CALL :ReplaceXMLTags ScriptPath
CALL :ReplaceXMLTags ScriptArgs
CALL :ReplaceXMLTags StartTime
CALL :ReplaceXMLTags Log
CALL :ReplaceXMLTags EndTime
CALL :ReplaceXMLTags ExecDuration 
CALL :ReplaceXMLTags ExecDurationInSec
CALL :ReplaceXMLTags TestStatus
CALL :ReplaceXMLTags TestsFailedCount
CALL :ReplaceXMLTags Duration
:: Change all "<" and ">" signs in the XML document.
findstr /r /c:"<.*>" "%logFilePath%" | "%actualPath%\fart.exe" -r -i "%logFilePath%" ^< {
findstr /r /c:"{.*>" "%logFilePath%" | "%actualPath%\fart.exe" -r -i "%logFilePath%" ^> }
CALL :RollbackXMLTags TestResult
CALL :RollbackXMLTags Test
CALL :RollbackXMLTags ScriptName
CALL :RollbackXMLTags ScriptPath
CALL :RollbackXMLTags ScriptArgs
CALL :RollbackXMLTags StartTime
CALL :RollbackXMLTags EndTime
CALL :RollbackXMLTags ExecDuration
CALL :RollbackXMLTags ExecDurationInSec
CALL :RollbackXMLTags Log
CALL :RollbackXMLTags TestStatus
CALL :RollbackXMLTags TestsFailedCount
CALL :RollbackXMLTags Duration
:: Indent de XML document with 3 spaces.
"%actualPath%\xml.exe" fo -s 3 "%logFilePath%">> "%logFilePath%.TEMP"
DEL /Q "%logFilePath%"
RENAME  "%logFilePath%.TEMP" %LogFileName%
GOTO:EOF



:: Replace the valid XML <tags> to {tags}, so they won't be affect when the "<" and ">" signs are replaced by "{" and "}".
:ReplaceXMLTags
"%actualPath%\fart.exe" -r -i "%logFilePath%" ^<%1^> {%1}
"%actualPath%\fart.exe" -r -i "%logFilePath%" ^</%1^> {/%1}
GOTO:EOF



:: Rollback the valid XML {tags} to <tags>.
:RollbackXMLTags
"%actualPath%\fart.exe" -r -i "%logFilePath%" {%1} ^<%1^>
"%actualPath%\fart.exe" -r -i "%logFilePath%" {/%1} ^</%1^>
GOTO:EOF

-- 
You received this question notification because you are a member of
Sikuli Drivers, which is an answer contact for Sikuli.