Skip to main content

Interacting with apps

Argent gives the agent control of the device. The agent taps, swipes, pinches and types on mobile targets. The agent presses the remote on TV targets. The agent clicks, scrolls and drags on desktop and web targets. The agent uses your app in the same way as a user.

The agent describes the screen, taps a photo, then pinches to zoom in on one person

One tool set for all platforms​

The agent uses the same tools on iOS, Android, TV, Chromium and Mac apps. Argent reads the device id and sends each action to the correct platform. Coordinates are fractions of the screen, from 0.0 to 1.0. Thus the same instruction works on screens of different sizes. On a Mac app, coordinates are fractions of the app window with its sheets, alerts and open menus.

Discovery before interaction​

The agent does not read coordinates from a screenshot. Before a tap, the agent has the elements on the screen with their roles, labels and positions. Each interaction tool returns these elements together with the screenshot, or names the element tree that the agent already has. The agent reads the screen separately only before the first action on a new screen. The source depends on the app:

  • the accessibility tree on iOS, Android, TV and Mac apps,
  • the DOM in Chromium and Electron apps.

After a screen change, the action returns the new screen: a screenshot and the element tree. The element tree also contains the hint of describe, for example when the tree is not complete. The next tap needs no extra call. Each element tree has an id. When the element tree is identical to the last one, Argent sends a short note with that id in its place, and the agent uses the element tree that it already has. The disable-auto-describe flag stops the element tree.

Launching apps​

The agent opens an app by its bundle id or package name. The agent does not tap an icon on the Home screen. Thus each run starts from a known state. The agent also opens deep links by URL, restarts an app, and reinstalls an app with clean data. On an Apple simulator, a physical iPhone or a Mac app, the agent can also give launch arguments to the app, for example to turn on a feature flag.

Gestures and text​

The agent taps, swipes, pinches, rotates content, and sends custom touch sequences for long press or drag and drop. On Chromium, the agent scrolls with the mouse wheel and drags with the mouse. On a Mac app, the agent drags, and scrolls the content under a point. The pointer of the Mac does not move. The agent taps a text field first, then types text and special keys such as enter or escape. When a user would use paste, the agent puts the text on the device clipboard and pastes it in one step. On a Mac app, Argent inserts the text without the clipboard of the Mac.

To type a credential, the agent uses a secret placeholder. Argent fills the value from your environment. The value does not enter the context of the agent. When Device Hub has opened a simulator, Argent types a secret instead of pasting it. The secret does not go to the Mac clipboard.

Permissions​

The agent grants, denies or resets a runtime permission such as location, camera or contacts. The agent does not navigate the Settings app. Then the agent launches the app again and sees the permission prompt. Thus a first-run permission flow is repeatable.

The agent resets the location permission, then relaunches the app to trigger the prompt again

Device state and hardware​

The agent also controls the device around the app. The agent rotates the device between portrait and landscape. The agent folds and unfolds a foldable iOS simulator or Android emulator. Screenshots, element trees and taps use the panel that shows the UI. The agent presses hardware buttons such as home, back and volume. On TV targets, the agent presses the remote. The agent shakes the device to start the handler that your app binds to a shake, for example a feedback sheet or the developer menu.

The agent reads what shake is wired to in the code, then shakes the simulator to test it

Waiting and sequences​

The agent does not take screenshots in a loop. The agent waits for a condition on the UI, such as "this element is visible" or "the screen stopped changing". When the wait for an element ends, Argent returns the screen: a screenshot and the element tree.

If the agent knows several actions before the start, and there is nothing to examine between them, the agent runs them as one sequence. One call replaces several round trips.

Example instructions​

  • "Open the app and go to the settings screen."
  • "Scroll the list to the end. Tell me the last item."
  • "Type hello in the search field. Press enter."
  • "Reset the camera permission and open the camera screen again."
  • "Rotate the device to landscape and make a screenshot."

Limits​

  • TV targets are focus-driven. The agent does not use touch gestures on them.
  • Swipe is for touch targets. On Chromium and on Mac apps, the agent scrolls and drags with the mouse.
  • Shake works on simulators and emulators only. Paste works on simulators, emulators and Mac apps.
  • On Chromium, each action goes to the active tab.

The tools reference contains all tools, including the ones in the Interacting and Devices and apps groups.