Sikuli : Commands used in Sikuli IDE

In our last post on Sikuli we have shown how we can automate a desktop application using Sikuli IDE with image recognition technique. Here we will discuss about the command used in Sikuli IDE.

Command List : Command list is displayed at left side panel. With the help of these commands we will write the scripts. There are four sections in the command list.

1. Settings : By checking Auto capture options can be enabled or not.
2. Find : These Command lists help the tester to write script to find an UI object.
3. Mouse Actions : All the commands is this section are related mouse actions.
4. Keyboard Actions : All the Keyboard related commands are there.

1. Settings : If Auto Capture option is uncheck then Sikuli IDE will not go to Screen Capture when click any command name from Command list. 

For example if we click on click() command after unchecking the Auto Capture option, it just displays the command name at the script editor like the following,

  

Note that a camera icon is displayed at the click(camera icon) command as argument. By clicking on camera icon Sikuli IDE will move to screen capture mode and then UI can be captured.

If we click on left mouse button at screen capture mode we can back to Sikuli IDE without capturing any screen UI object.

If Auto Capture option is checked and click on a command then it automatically goes to screen capture mode and a text displays on the screen “Select an image”.

Clicking on Expand/Collapsed button, commands section can be expanded and collapsed.

2. Find : In this section, all the Sikuli IDE command related to find an UI objects are displayed. There are five commands here.

• find(captured image) : It finds out the given captured image on the screen. General syntax of find() command.

find(captured image)

For example,

 

• findAll(captured image) : It return all the available locations of the given captured image on the screen. If there are multiple UI images on the screen then we can use this findAll() and gets all the locations of the given images. We can use loop structure to access all the images. Note that Sikuli script editor supports python language.

• wait(captured image) : It waits for the given image to appear. It takes two arguments- one is the captured image and other is the time to wait. Time is counted as millisecond. General Syntax,

wait(Captured image, timestamp)

Note that we can use the constant “FOREVER” in place of timestamp that waits for the captured images up to indefinite time.

 

• waitVanish(captured image, timestamp) : It waits for the given image to vanish. The same syntax here as wait(),

waitVanish(Captured image, timestamp)

• exists() : It returns the matched UI object that is given as an argument. It the captured image given as an argument it checks and returns match object if it is matched on the screen. Generally we use it to verify expected value on our scripting. General syntax,

exists(Captured image, [timestamp])

Note that here timestamp is the time to wait for the image to appear and it is optional. We can set default time.

3. Mouse Actions : Sometimes we need to click on a button, check/uncheck a check box or option button or set focus on a text box, all of the cases we have to handle mouse event. Commands on this section will perform this activities.

• click(captured image) : It is used to click on a captured image. General syntax,

click(Captured image)

For example, if we want to click on a button on our calculator application, command will be,

Note that here click means mouse’s left button click.

• doubleClick(captured image) : It is used to double click on a captured image. General syntax,

doubleClick(Captured image)

Note that here click means mouse’s left button click.

• rightClick(captured image) : Clicking by Right mouse button on a captured image. General syntax,

rightClick(Captured image)

• hover(captured image) : If we want to hover mouse pointer on an UI object then we use hover() command. General syntax,

hover(Captured image) 

Note that it returns 1 if it finds the UI image similar to captured image or 0 if it does not.

• dragDrop(source image, destination image) : It is used to drag an object from a location and drop it to other location. General syntax,

dragDrop(source image, destination image)

4. Keyboard Actions : In order to enter data as input in a text box or copy/paste input data we need to handle keyboard actions and in that we have to write keyboard command. In Sikuli, following commands are used.

• type(text) : If we want to type some text on a text box we can use this command. It takes text as argument. Most of the cases we click on a text box to set focus and then type text. General Syntax,

type(text)

Note that it supports all keyboard commands like TAB, ESC etc.

• type(captured image, text) : This command first finds the captured image and then type the text it that area. General Syntax,

type(captured image, text)

 

• paste(text) : If we want to paste a text to a location then we can use this command. General Syntax,

paste(text)

• paste(captured image, text) : If we want to paste text to a desired location that is captured by an image we can use this paste() command. general syntax,

paste(captured image, text)

Here captured image is the location where given text is pasted.

Happy Scripting by capturing UI image using Sikuli….!!!