Sikuli : How to automate an application using Sikuli IDE

In our last post on Sikuli we have shown how to install Sikuli in a Windows Machine. Here we see how we can automate a desktop application using Sikuli IDE with image recognition technique.

At First we need to write a test case we have to automate using Sikuli. In our case, we will use our built in Calculator application.

Test Case : Verify the add functionality of our calculator application.

1. Open the application
2. Click on (7) seven button
3. Click on plus (+) button
4. Click on (3) three button
5. Click on (=) equals button
6. Verify the result

Step 1 : Go to installation directory where our Sikuli is installed. In my case it is “C:\Sikuli". Those who follow my last post on Sikuli can view the same location.

Step 2 : Double click on the file name “sikuli-ide.jar”. You can create a shortcut of this file to show your desktop.

Step 3 : Runtime options window displayed and click on OK button.

Step 4 : Sikuli IDE window is displayed.

Step 5 : We need an introduction on the IDE.

Title Bar : It displays the application name “Sikuli IDE 1.0.1” with the Script file name. Default name is “Untitled”.

Menu Bar : It contains all the available menus.

1. File : All the basic options are there. There is a Preference option to set the Sikuli settings.
2. Edit : All the basic options are there.
3. Run : There are two options – Run and Run in Slow motion. If you want to run script slowly you can use this “Run in Slow motion” option.
4. View : View the command list and menu.
5. Tools : All the tools option.
6. Help : Sikuli help and FAQ document.

Tool Bar : It contains Six options.

1. Take screenshot : Clicking on this button Sikuli starts to capture the image of an UI object bye which we will use it for scripting.
2. Insert image : Through this option we can insert image from other location captured by other image captured tool.
3. Create Region : We can create a region to help Sikuli to search the UI object to within this region by this option.
4. Run : By clicking this button we can run the Sikuli script written at the Below editor.
5. Run in Slow Motion : By using this option we can run the script slowly.
6. Find : Find some text within the script.

Command List : Command list is displayed at left side panel. With the help of these commands we will write the scripts.

1. Settings : By checking Auto capture options can be enabled or not.
2. Find : These Command lists help the tester to write script to find an UI object.
3. Mouse Actions : All the commands is this section are related mouse actions.
4. Keyboard Actions : All the Keyboard related commands are there.

Note that we will discuss more details on command list in a separate blog post.

Step 6 : Now start writing the script using Sikuli IDE.

• At first we need to open the application. For this we need to click the calculator icon located  at the task bar. Now click on “click” command from left side command list.

• Sikuli IDE will be disappeared and screen will be fade. A text “Select an image” is displayed at desktop and mouse icon changed to cross (+).

 

• Now drag the mouse pointer on the calculator icon to capture the whole area like the following.

 

• Releasing mouse left button appears the Sikuli IDE and command is displayed at the sikuli editor.

 

• Note that “click()” command is displayed with captured image as argument.

Common syntax of a command,

commandName(captured Image)

commandName : it is the name of the command like “Find()”, “click()” etc.

captured image : You can capture image using IDE or any tool.

• Click on (7) seven button. For this step, go to command list and click on “click()” command and drag the mouse pointer over the button “7”. It captured the image of Seven(7) button and displayed it to the Sikuli script editor like the following.

• Click on plus (+) button. Do the same as we have done for seven (7) button.

• Click on (3) three button

• Click on (=) equals button

   

• Verify the result. In order to verify the result, click on exists() command and drag the mouse pointer to result area and the command will look like the following.

    

Note that exists() commands compare the image with the captured image.

After that the script will look like as the following,

Step 7 : Save the script by clicking File >> Save.

Step 8 : Enter the script name “CalculatorTest” at the File save Text box. Note that it has default file extension “*.sikuli”. Click on Save button. It has been saved to a desired location.

Step 9 : Run the script by Clicking Run button on the Toolbar.

The script will run. It opens the calculator by clicking on toolbar icon, then click on 7,+,3,= buttons and then verify the image with the result box. After Run the Sikuli IDE will look like the following,

Note that you can use Run in Slow motion button to run the script slowly.

That’s all for now. We will do more discussion on our future post on Sikuli and its future.

Happy Scripting by capturing UI image using Sikuli….!!!