Execute Computer Use or Mobile Use action
Executes a computer use or mobile use action on the sandbox.
Path Parameters
The organization ID
All possible computer use actions, with optional callId
Click the mouse at the specified coordinates
"mouse:click"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Mouse button flag combination. 1: left, 2: right, 4: middle, 8: back, 16: forward; add them together to press multiple buttons at once.
Key to hold down during click, in xdotool key syntax. Example: "ctrl", "alt", "alt+shift".
Whether the coordinates are relative to the current mouse position.
Double click the mouse at the specified coordinates
"mouse:doubleClick"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Mouse button index
Key to hold down during double click, in xdotool key syntax. Example: "ctrl", "alt", "alt+shift".
Whether the coordinates are relative to the current mouse position.
Triple click the mouse at the specified coordinates
"mouse:tripleClick"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Mouse button index
Key to hold down during triple click, in xdotool key syntax. Example: "ctrl", "alt", "alt+shift".
Whether the coordinates are relative to the current mouse position.
Move the mouse to the specified coordinates
"mouse:move"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Key to hold down during move, in xdotool key syntax. Example: "ctrl", "alt", "alt+shift".
Whether the coordinates are relative to the current mouse position.
Scroll the mouse
"mouse:scroll"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Vertical scroll steps
Horizontal scroll steps
Key to hold down during scroll, in xdotool key syntax. Example: "ctrl", "alt", "alt+shift".
Whether the coordinates are relative to the current mouse position.
Drag the mouse from start to end coordinates
"mouse:drag"Start X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Start Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
End X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
End Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Mouse button index
Key to hold down during drag, in xdotool key syntax. Example: "ctrl", "alt", "alt+shift".
Whether the coordinates are relative to the current mouse position.
Whether the coordinates are relative to the current mouse position.
Type text content
"keyboard:type"Text content to type
Whether to treat line breaks as enter. If true, any line breaks(\n) in content will be treated as enter key press, and content will be split into multiple lines.
falsePress a keyboard hotkey
"keyboard:hotkey"Hotkey combination, in xdotool key syntax. Examples: "a", "Return", "alt+Tab", "ctrl+s", "Up", "KP_0" (for the numpad 0 key).
Duration in milliseconds. If specified, the hotkey will be held for a while and then released.
1 <= value <= 5000Take a screenshot of the current screen
"screenshot"Wait for a specified duration
"wait"Duration in milliseconds
Indicates the action has finished
"finished"Completion message
Indicates the action has failed
"failed"Failure message
Indicates the human user should take over the control
"client:user-takeover"Press ONE key down, in xdotool key syntax. Only use this action if hotkey or type cannot satisfy your needs.
"key:down"Key to press
Release ONE key, in xdotool key syntax. Only use this action if keydown cannot satisfy your needs and only after a key down.
"key:up"Key to release
All possible mobile use actions, with optional callId
Take a screenshot of the current screen
"screenshot"Wait for a specified duration
"wait"Duration in milliseconds
Indicates the action has finished
"finished"Completion message
Indicates the action has failed
"failed"Failure message
Indicates the human user should take over the control
"client:user-takeover"Type text content
"keyboard:type"Text content to type
Whether to treat line breaks as enter. If true, any line breaks(\n) in content will be treated as enter key press, and content will be split into multiple lines.
falsePress a keyboard hotkey
"keyboard:hotkey"Hotkey combination, in xdotool key syntax. Examples: "a", "Return", "alt+Tab", "ctrl+s", "Up", "KP_0" (for the numpad 0 key).
Duration in milliseconds. If specified, the hotkey will be held for a while and then released.
1 <= value <= 5000Tap the screen at the specified coordinates
"touch:tap"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Touch and hold at start coordinates, then move to end coordinates and release
"touch:drag"Start X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Start Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
End X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
End Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Swipe the screen in a specified direction by a specified distance
"touch:swipe"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Scroll direction
"up" | "down" | "left" | "right"Scroll distance
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Long press the screen at the specified coordinates
"touch:longPress"X coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Y coordinate
Length in pixels
Indicates the unit is pixel
"px"Length in pixels
Length as a fraction
"/"Numerator of the fraction
Denominator of the fraction
Duration in milliseconds
Press the back button
"android:back"Press the home button
"android:home"List all installed apps
"os:listApps"Start an app by its package name
"os:startApp"App package name
Start an app by its name
"os:startAppByName"App name
Close the current app
"os:closeApp"Whether to include the screenshot url after action in the response
trueWhether to include the cursor position after action in the response. On some cursor-less devices, only width and height are meaningful.
trueResponse Body
{
"screenShot": "http://example.com",
"cursorPosition": {
"x": 0,
"y": 0,
"screenWidth": 0,
"screenHeight": 0,
"screenIndex": 0
},
"actionResult": null
}