Sikuli : Commands used in Sikuli IDE

In our last post on Sikuli we have shown how we can automate a desktop application using Sikuli IDE with image recognition technique. Here we will discuss about the command used in Sikuli IDE.

Command List : Command list is displayed at left side panel. With the help of these commands we will write the scripts. There are four sections in the command list.

1. Settings : By checking Auto capture options can be enabled or not.
2. Find : These Command lists help the tester to write script to find an UI object.
3. Mouse Actions : All the commands is this section are related mouse actions.
4. Keyboard Actions : All the Keyboard related commands are there.

1. Settings : If Auto Capture option is uncheck then Sikuli IDE will not go to Screen Capture when click any command name from Command list. 

For example if we click on click() command after unchecking the Auto Capture option, it just displays the command name at the script editor like the following,

  

Note that a camera icon is displayed at the click(camera icon) command as argument. By clicking on camera icon Sikuli IDE will move to screen capture mode and then UI can be captured.

If we click on left mouse button at screen capture mode we can back to Sikuli IDE without capturing any screen UI object.

If Auto Capture option is checked and click on a command then it automatically goes to screen capture mode and a text displays on the screen “Select an image”.

Clicking on Expand/Collapsed button, commands section can be expanded and collapsed.

2. Find : In this section, all the Sikuli IDE command related to find an UI objects are displayed. There are five commands here.

• find(captured image) : It finds out the given captured image on the screen. General syntax of find() command.

find(captured image)

For example,

 

• findAll(captured image) : It return all the available locations of the given captured image on the screen. If there are multiple UI images on the screen then we can use this findAll() and gets all the locations of the given images. We can use loop structure to access all the images. Note that Sikuli script editor supports python language.

• wait(captured image) : It waits for the given image to appear. It takes two arguments- one is the captured image and other is the time to wait. Time is counted as millisecond. General Syntax,

wait(Captured image, timestamp)

Note that we can use the constant “FOREVER” in place of timestamp that waits for the captured images up to indefinite time.

 

• waitVanish(captured image, timestamp) : It waits for the given image to vanish. The same syntax here as wait(),

waitVanish(Captured image, timestamp)

• exists() : It returns the matched UI object that is given as an argument. It the captured image given as an argument it checks and returns match object if it is matched on the screen. Generally we use it to verify expected value on our scripting. General syntax,

exists(Captured image, [timestamp])

Note that here timestamp is the time to wait for the image to appear and it is optional. We can set default time.

3. Mouse Actions : Sometimes we need to click on a button, check/uncheck a check box or option button or set focus on a text box, all of the cases we have to handle mouse event. Commands on this section will perform this activities.

• click(captured image) : It is used to click on a captured image. General syntax,

click(Captured image)

For example, if we want to click on a button on our calculator application, command will be,

Note that here click means mouse’s left button click.

• doubleClick(captured image) : It is used to double click on a captured image. General syntax,

doubleClick(Captured image)

Note that here click means mouse’s left button click.

• rightClick(captured image) : Clicking by Right mouse button on a captured image. General syntax,

rightClick(Captured image)

• hover(captured image) : If we want to hover mouse pointer on an UI object then we use hover() command. General syntax,

hover(Captured image) 

Note that it returns 1 if it finds the UI image similar to captured image or 0 if it does not.

• dragDrop(source image, destination image) : It is used to drag an object from a location and drop it to other location. General syntax,

dragDrop(source image, destination image)

4. Keyboard Actions : In order to enter data as input in a text box or copy/paste input data we need to handle keyboard actions and in that we have to write keyboard command. In Sikuli, following commands are used.

• type(text) : If we want to type some text on a text box we can use this command. It takes text as argument. Most of the cases we click on a text box to set focus and then type text. General Syntax,

type(text)

Note that it supports all keyboard commands like TAB, ESC etc.

• type(captured image, text) : This command first finds the captured image and then type the text it that area. General Syntax,

type(captured image, text)

 

• paste(text) : If we want to paste a text to a location then we can use this command. General Syntax,

paste(text)

• paste(captured image, text) : If we want to paste text to a desired location that is captured by an image we can use this paste() command. general syntax,

paste(captured image, text)

Here captured image is the location where given text is pasted.

Happy Scripting by capturing UI image using Sikuli….!!!

Sikuli : Introduction to Sikuli IDE

In our last post on Sikuli we have shown how we can automate a desktop application using Sikuli IDE with image recognition technique. There we have shown the steps to automate a desktop application and discussed about the Sikuli IDE. Here we will discuss more details on it.

Sikuli IDE is noting but an IDE where we can build Sikuli script and run it.

 

Title Bar : It displays the application name “Sikuli IDE 1.0.1” with the Script file name. Default name is “Untitled”.

Menu Bar : It contains all the available menus.

1. File : All the basic options are there. There is a Preference option to set the Sikuli settings.
2. Edit : All the basic options are there.
3. Run : There are two options – Run and Run in Slow motion. If you want to run script slowly you can use this “Run in Slow motion” option.
4. View : View the command list and menu.
5. Tools : All the tools option.
6. Help : Sikuli help and FAQ document.

Tool Bar : It contains Six options.

1. Take screenshot : Clicking on this button Sikuli starts to capture the image of an UI object bye which we will use it for scripting.
2. Insert image : Through this option we can insert image from other location captured by other image captured tool.
3. Create Region : We can create a region to help Sikuli to search the UI object to within this region by this option.
4. Run : By clicking this button we can run the Sikuli script written at the Below editor.
5. Run in Slow Motion : By using this option we can run the script slowly.
6. Find : Find some text within the script.

Command List : Command list is displayed at left side panel. With the help of these commands we will write the scripts.

1. Settings : By checking Auto capture options can be enabled or not.
2. Find : These Command lists help the tester to write script to find an UI object.
3. Mouse Actions : All the commands is this section are related mouse actions.
4. Keyboard Actions : All the Keyboard related commands are there.

Note that we will discuss more details on command list in a separate blog post.



Script Editor : All the scripts are written here and user can modify it. Note that Sikuli IDE support python language syntax. So we can write our python code here and make the script more robust and maintainable and implement different programming techniuqe using the power of python programming language.

That’s all for now. We will do more discussion on our future post on Sikuli and its future.

Happy Scripting by capturing UI image using Sikuli….!!!

Sikuli : How to automate an application using Sikuli IDE

In our last post on Sikuli we have shown how to install Sikuli in a Windows Machine. Here we see how we can automate a desktop application using Sikuli IDE with image recognition technique.

At First we need to write a test case we have to automate using Sikuli. In our case, we will use our built in Calculator application.

Test Case : Verify the add functionality of our calculator application.

1. Open the application
2. Click on (7) seven button
3. Click on plus (+) button
4. Click on (3) three button
5. Click on (=) equals button
6. Verify the result

Step 1 : Go to installation directory where our Sikuli is installed. In my case it is “C:\Sikuli". Those who follow my last post on Sikuli can view the same location.

Step 2 : Double click on the file name “sikuli-ide.jar”. You can create a shortcut of this file to show your desktop.

Step 3 : Runtime options window displayed and click on OK button.

Step 4 : Sikuli IDE window is displayed.

Step 5 : We need an introduction on the IDE.

Title Bar : It displays the application name “Sikuli IDE 1.0.1” with the Script file name. Default name is “Untitled”.

Menu Bar : It contains all the available menus.

1. File : All the basic options are there. There is a Preference option to set the Sikuli settings.
2. Edit : All the basic options are there.
3. Run : There are two options – Run and Run in Slow motion. If you want to run script slowly you can use this “Run in Slow motion” option.
4. View : View the command list and menu.
5. Tools : All the tools option.
6. Help : Sikuli help and FAQ document.

Tool Bar : It contains Six options.

1. Take screenshot : Clicking on this button Sikuli starts to capture the image of an UI object bye which we will use it for scripting.
2. Insert image : Through this option we can insert image from other location captured by other image captured tool.
3. Create Region : We can create a region to help Sikuli to search the UI object to within this region by this option.
4. Run : By clicking this button we can run the Sikuli script written at the Below editor.
5. Run in Slow Motion : By using this option we can run the script slowly.
6. Find : Find some text within the script.

Command List : Command list is displayed at left side panel. With the help of these commands we will write the scripts.

1. Settings : By checking Auto capture options can be enabled or not.
2. Find : These Command lists help the tester to write script to find an UI object.
3. Mouse Actions : All the commands is this section are related mouse actions.
4. Keyboard Actions : All the Keyboard related commands are there.

Note that we will discuss more details on command list in a separate blog post.

Step 6 : Now start writing the script using Sikuli IDE.

• At first we need to open the application. For this we need to click the calculator icon located  at the task bar. Now click on “click” command from left side command list.

• Sikuli IDE will be disappeared and screen will be fade. A text “Select an image” is displayed at desktop and mouse icon changed to cross (+).

 

• Now drag the mouse pointer on the calculator icon to capture the whole area like the following.

 

• Releasing mouse left button appears the Sikuli IDE and command is displayed at the sikuli editor.

 

• Note that “click()” command is displayed with captured image as argument.

Common syntax of a command,

commandName(captured Image)

commandName : it is the name of the command like “Find()”, “click()” etc.

captured image : You can capture image using IDE or any tool.

• Click on (7) seven button. For this step, go to command list and click on “click()” command and drag the mouse pointer over the button “7”. It captured the image of Seven(7) button and displayed it to the Sikuli script editor like the following.

• Click on plus (+) button. Do the same as we have done for seven (7) button.

• Click on (3) three button

• Click on (=) equals button

   

• Verify the result. In order to verify the result, click on exists() command and drag the mouse pointer to result area and the command will look like the following.

    

Note that exists() commands compare the image with the captured image.

After that the script will look like as the following,

Step 7 : Save the script by clicking File >> Save.

Step 8 : Enter the script name “CalculatorTest” at the File save Text box. Note that it has default file extension “*.sikuli”. Click on Save button. It has been saved to a desired location.

Step 9 : Run the script by Clicking Run button on the Toolbar.

The script will run. It opens the calculator by clicking on toolbar icon, then click on 7,+,3,= buttons and then verify the image with the result box. After Run the Sikuli IDE will look like the following,

Note that you can use Run in Slow motion button to run the script slowly.

That’s all for now. We will do more discussion on our future post on Sikuli and its future.

Happy Scripting by capturing UI image using Sikuli….!!!

Sikuli : How to install and run Sikuli in windows 7

Sikuli is an automated testing tool where one can automate the workflow of the application by using image of an element of UI and searching the image with the help of image detection algorithm. I have already discussed about Sikuli in my previous post. Here I will explain on how Sikuli can be installed and run and what are the pre-requisites we need.

Requirement :

Sikuli can be installed at the following OS and platforms.

• Windows Xp/7/8 both 32 bit and 64 bit
• Mac OS 10.6+
• Linux 32 bit or 64 bit

Pre-Requisite :

Java JRE 6 or JRE 7 preferably the latest version.

Step 1: Install the pre-Requisite

• Download Java if it is not installed in the PC. Latest Java (JRE) can be downloaded from this link. Note that only JRE 6 or 7 supports Sikuli. Sikuli works well with Windows 7 32 bit OS.
• After going to Java download page click on “Accept License Agreement” button.

• After clicking License agreement button, it shows the Thank you message.
• Now click on appropriate Java installer for the machine. In my case, I have selected on “Windows x86 Offline” for my Windows 7 32 bit machine.

• After downloading, double click on the installer “jre-7-windows-i586.exe”.
• Click on Run button.

• Click on Install button.

• Installation will start.

• Java runtime will be successfully installed. Click on Close button.

• To verify that Java is working properly, go to the link : https://www.java.com/en/download/installed.jsp
• Click on “Agree and Continue” button

• Sometimes a pop up can be appeared.
• Click on Run button

• A message shows that Java is installed properly.

Step 2 : Download Sikuli

Latest and stable version of Sikuli is 1.0.1. Version SikuliX 1.1.0 is final under development.

• Go to the page : http://www.sikuli.org/download.html
• Click on the link “sikuli-setup.jar”

• A page is displayed with the link https://launchpad.net/sikuli/+download
• click on “sikuli-setup.jar” link.

• Sometimes an alert message is shown as I am using chrome browser. click on “Keep” button.

• Download will start.

 

Step 3 :  Install Sikuli

• After Finishing download go to the download folder and copy the installer file and paste to a location when you install the file. In my case file location “C:\Sikuli\”
• Double click on the installer

• An instruction dialog is displayed that instructs to go to the File location and run the file named “runSetup.cmd” and click on OK button.

• Go to the installer file location : “C:\Sikuli\”. Following files are displayed.

• Double click on the file “runSetup.cmd” to run.

`

• A  command prompt is displayed. After that a dialog box is displayed with the text “Please read Carefully before proceedings!!”. Click on Ok button.

• SikuliX-Setup window is displayed.

• There are six options displayed on this window.

Option 1 :

If you choose this option you will be able to use sikuli IDE (discuss on other posts) and run the Sikuli scripts through command line. Using Sikuli jars you can implement sikuli script with the help of Java or Java aware scripting language and gets supports from Eclipse or NetBeans IDE. Clicking rightmost “…” button will display a dialog showing the details of choosing this option. Click OK button will close this dialog.

   

Option 2 :

If you choose this option, you can only be able to run the script through command line. Clicking rightmost “…” button will display a dialog showing the details of choosing this option. Click OK button will close this dialog. If Option 1 is selected with this option then you need to run setup again only selecting this option. So the primary objective of this package to run Sikuli script on command line and no need Sikuli IDE.

 

Option 3 :

If you want to develop and run script in Jython language using Eclipse IDE then  you can choose this option. Clicking rightmost “…” button will display a dialog showing the details of choosing this option. Click OK button will close this dialog. Note that you can choose Option 1 or 2 with this option.

Option 4 : 

If you want to develop sikuli script in Java or Jython using Eclipse/NetBeans or other IDE and you do not need option 1 or 2 you can use this option. Clicking rightmost “…” button will display a dialog showing the details of choosing this option. Click OK button will close this dialog.

 

Option 5 :

  

If you want to Tesseract based OCR features you can this option. It is recommended not to use this option if you are new to Sikuli or using Sikuli for the first time. Clicking rightmost “…” button will display a dialog showing the details of choosing this option. Click OK button will close this dialog.

Option 6 : 

To run the packages (containing Sikuli script) on all supported platforms (Windows, Mac and Linux), you can choose this option. Clicking rightmost “…” button will display a dialog showing the details of choosing this option. Click OK button will close this dialog.

• In my case I have selected Option 1, 2 and 4. Click on Setup Now button.

• A confirmation dialog is displayed to ask about downloading some of the packages. Click on Yes button.

• Downloading starts…

• Some yellow color status text is displayed during installation.

• A status dialog is displayed after installation that it is installed successfully. Click OK button to close the dialog.

• If you go back to the Sikuli installation folder (In my case “C:\Sikuli”), you see some additional files are added and a “libs” folder.

 

Step 4 : Run Sikuli IDE

• In Sikuli installation folder, double click on the file named “runIDE.cmd”.

• Sikuli IDE is displayed and we can develop and run script using this IDE.

In our future posts, we will show how we can develop script in Sikuli IDE and run it.

Happy Scripting by capturing UI image using Sikuli….!!!