GitHub - testingbot/testingbotctl: A command line interface for the TestingBot platform. Run Maestro, XCUITest and Espresso tests.

Run mobile tests on real devices in the cloud

Run Espresso, XCUITest and Maestro tests on real devices in the cloud.

Real Devices — Test on thousands of real iOS and Android devices
Emulators & Simulators — Fast feedback with virtual devices
Parallel Execution — Split tests across multiple devices with sharding
CI/CD Ready — Integrates with GitHub Actions, Jenkins, and more
Live Results — Watch tests run in real-time
Artifacts — Download videos, screenshots, and logs

Installation

npm install -g @testingbot/cli

Requirements: NodeJS 20 or higher

Authentication

The CLI requires TestingBot API credentials. You can authenticate in several ways:

Browser Login (Recommended)

testingbot login

This opens your browser for authentication. After logging in, your credentials are saved to ~/.testingbot.

Other Methods

Command-line options: --api-key and --api-secret
Environment variables: TB_KEY and TB_SECRET
Config file: Create ~/.testingbot with content key:secret

Commands

Maestro

Run Maestro UI tests on real devices and emulators/simulators.

testingbot maestro <app> <flows...> [options]

Arguments:

app - Path to your app file (.apk, .ipa, .app, or .zip)
flows - One or more paths to flow files (.yaml/.yml), directories, .zip files, or glob patterns

App Options:

Option	Description
`--app <path>`	Path to the application under test (alternative to the positional `app` argument)
`--other-app <path-or-url>`	Additional companion app to install on the device alongside `--app`. Accepts a local file path (`.apk`, `.ipa`, `.app`, `.zip`) or a `tb://<appkey>` / `http(s)://...` URL — local paths are uploaded; URLs are passed through to the run as-is. Repeatable, max 4 entries.
`--app-binary-id <projectId>`	Reuse the app of a project uploaded earlier (`testingbot upload`, or any previous run's Project ID) instead of uploading one. Every positional argument is then a flow. The platform is taken from the stored app unless `--platform` is given

Device Options:

Option	Description
`--device <name>`	Device name (e.g., "Pixel 9", "iPhone 16")
`--platform <name>`	Platform: Android or iOS
`--deviceVersion <version>`	OS version (e.g., "14", "17.2")
`--real-device`	Use a real device instead of emulator/simulator
`--orientation <orientation>`	Screen orientation: PORTRAIT or LANDSCAPE
`--device-locale <locale>`	Device locale (e.g., "en_US", "de_DE")
`--timezone <timezone>`	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
`--name <name>`	Test name for dashboard identification
`--build <build>`	Build identifier for grouping test runs
`--groups <names>`	Tag the test session with one or more groups (comma-separated). Groups appear on the test in the TestingBot dashboard
`--include-tags <tags>`	Only run flows with these tags (comma-separated)
`--exclude-tags <tags>`	Exclude flows with these tags (comma-separated)
`-e, --env <KEY=VALUE>`	Environment variable for flows (can be repeated)
`--config <path>`	Path to a custom Maestro config file (default: config.yaml in project root)
`--maestro-version <version>`	Maestro version to use (e.g., "2.0.10")

Network & Location:

Option	Description
`--throttle-network <speed>`	Network throttling: 4G, 3G, Edge, airplane, or disable
`--geo-country-code <code>`	Geographic IP location (ISO country code, e.g., "US", "DE")

Tunnel:

Option	Description
`-t, --tunnel`	Start a TestingBot tunnel for this test run (cannot be combined with `--async`)
`--tunnel-identifier <id>`	Identifier for the tunnel, allowing multiple tunnels in parallel

Output Options:

Option	Description
`--async`	Start tests and exit without waiting for results
`-q, --quiet`	Suppress progress output
`--json`	Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail
`--json-file`	Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file
`--json-file-name <path>`	Custom path for the JSON results file (requires `--json-file`)
`--report <format>`	Download report after completion: html or junit
`--report-output-dir <path>`	Directory to save reports (required with --report)
`--download-artifacts [mode]`	Download test artifacts (logs, screenshots, video). Mode: `all` (default) or `failed`
`--artifacts-output-dir <path>`	Directory to save artifacts zip (defaults to current directory)

Advanced Options:

Option	Description
`--shard-split <number>`	Split flows into N parallel sessions for faster execution
`--retry <count>`	Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`.
`--ignore-checksum-check`	Skip checksum verification and always upload the app

Note on --retry: a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a ↻ icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the last attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via --report.

CI/CD Integration:

Option	Description
`--commit-sha <sha>`	Git commit SHA associated with this test run
`--pull-request-id <id>`	Pull request ID this test run originated from
`--repo-name <name>`	Repository name (e.g., GitHub repo slug)
`--repo-owner <owner>`	Repository owner (e.g., GitHub organization or username)

Examples:

# Basic usage
testingbot maestro app.apk ./flows

# Multiple flow directories
testingbot maestro app.apk ./flows/smoke ./flows/regression ./flows/e2e

# With device selection
testingbot maestro app.apk ./flows --device "Pixel 8" --deviceVersion "14"

# Android app on real device with tags
testingbot maestro app.apk ./flows --device "Samsung Galaxy S24" --real-device --include-tags "smoke,regression"

# Tag the test session with groups (visible in the dashboard)
testingbot maestro app.apk ./flows --groups "smoke,critical"

# With environment variables
testingbot maestro app.apk ./flows -e API_URL=https://staging.example.com -e API_KEY=secret

# With companion apps installed alongside the main app (up to 4)
# Each --other-app can be a local file (uploaded) or a tb:// / http(s):// URL (passed through)
testingbot maestro --app main.apk \
  --other-app helper.apk \
  --other-app tb://existing-appkey \
  --other-app https://example.com/mock-server.apk \
  ./flows

# Download JUnit report
testingbot maestro app.apk ./flows --report junit --report-output-dir ./reports

# Download all artifacts (logs, screenshots, video)
testingbot maestro app.apk ./flows --download-artifacts --build "build-123"

# Download artifacts only for failed tests
testingbot maestro app.apk ./flows --download-artifacts failed --artifacts-output-dir ./artifacts

# Use a custom config file
testingbot maestro app.apk ./flows --config .maestro/ci-config.yaml

# Run in background (async)
testingbot maestro app.apk ./flows --async

# Split flows across 3 shards, grouping all flows over 3 parallel sessions
testingbot maestro app.apk ./flows --shard-split 3

# Retry failed flows up to 2 times (re-runs only the flows that failed)
testingbot maestro app.apk ./flows --retry 2

# CI/CD integration with Git metadata
testingbot maestro app.apk ./flows \
  --commit-sha "abc123def" \
  --pull-request-id "42" \
  --repo-owner "myorg" \
  --repo-name "myapp"

Organizing flows and subflows

Every top-level flow you pass runs as its own test. A subflow (a reusable flow another flow pulls in with runFlow) should not be passed as a top-level flow — if it is, it runs twice: once standalone and once as part of the flow that calls it.

Maestro has no notion of a "subflow-only" file. A .yaml sitting alongside your real flows is a runnable flow, regardless of its name. Naming it *.shared.yaml does not make Maestro treat it as shared.

Recommended structure — keep subflows in their own directory:

flows/
  login.yaml            # top-level, runs
  checkout.yaml         # top-level, runs
  subflows/
    sign-in.yaml        # only runs when a flow calls it via runFlow

# flows/login.yaml
- runFlow:
    file: subflows/sign-in.yaml
    env:
      APP_ID: com.example.app

Then pass only the directory of top-level flows:

# Runs login.yaml and checkout.yaml; sign-in.yaml is bundled automatically
# (as a runFlow dependency) but never runs on its own.
testingbot maestro app.apk ./flows

When you pass individual files, list only the flows you want to run — their runFlow targets are discovered and uploaded for you:

# Correct: only the top-level flow. sign-in.yaml is bundled automatically.
testingbot maestro app.apk ./flows/login.yaml

# Wrong: sign-in.yaml would run twice.
testingbot maestro app.apk ./flows/login.yaml ./flows/subflows/sign-in.yaml

Other ways to keep a subflow out of a run:

config.yaml globs — list only the folders that hold top-level flows (e.g. flows: ["*.yaml"]), leaving subflow folders out of discovery.
Tags — add tags: [subflow] to the subflow's header and pass --exclude-tags subflow.

Preview before you run. --dry-run prints exactly which flows run standalone and which are bundled as runFlow subflows, without spending any device minutes:

testingbot maestro app.apk ./flows --dry-run

testingbotctl also prints a warning if a flow you passed will run more than once because another top-level flow calls it via runFlow.

Upload once, run many times

testingbot upload pushes an app once and prints a Project ID. Later runs pass that ID with --app-binary-id and skip the upload entirely; each run still gets its own project and results.

testingbot upload app.apk
#   Uploaded app.apk. Project ID: 4321
#   Run flows against it with: testingbot maestro --app-binary-id 4321 ./flows

APP_ID=$(testingbot upload app.apk --json | jq -r .appId)
testingbot maestro --app-binary-id "$APP_ID" ./flows/smoke
testingbot maestro --app-binary-id "$APP_ID" ./flows/regression --device "Pixel 9"

Every maestro run also prints its Project ID after the app upload, so any previous run's ID works with --app-binary-id too. Unchanged binaries are deduplicated by checksum on upload as well; pass --ignore-checksum-check to force a fresh upload.

upload <appFile>

Option	Description
`--ignore-checksum-check`	Skip checksum verification and always upload the app
`-q, --quiet`	Suppress upload progress

--json returns { provider, appId, file, url }. Fails with exit code 1 if the upload was rejected.

Status, artifacts and list

Commands for working with Maestro projects after they were started, typically together with --async. Every command accepts --api-key / --api-secret, --debug, and the --json, --json-file, --json-file-name output flags described under JSON Output.

# Start tests without waiting and capture the project id
testingbot maestro app.apk ./flows --async --json | jq -r .appId

# Check on it later; --wait blocks with live progress and exits 2 on failure
testingbot status --id 1234
testingbot status --id 1234 --wait

# Fetch reports and artifacts once it finished
testingbot artifacts --id 1234 --report junit --report-output-dir ./reports
testingbot artifacts --id 1234 --download-artifacts failed --artifacts-output-dir ./artifacts

# Browse recent projects
testingbot list
testingbot list --count 25 --offset 25 --json

status --id <projectId>

Option	Description
`-w, --wait`	Block until every run has finished, showing the same live flow table as a foreground run
`-q, --quiet`	Suppress progress output

Exit code is 0 while the project is still running (JSON outcome: "running"), 0/2 once it completed, 1 on errors.

artifacts --id <projectId>

Option	Description
`--report <format>`	Download report: `html`, `html-detailed` or `junit`
`--report-output-dir <path>`	Directory to save reports (required with `--report`)
`--download-artifacts [mode]`	Download logs, screenshots and video. Mode: `all` (default) or `failed`
`--artifacts-output-dir <path>`	Directory to save the artifacts zip (defaults to current directory)

Fails with exit code 1 if the project is still running; use status --wait first.

list

Option	Description
`--count <number>`	Maximum number of projects to return (default 10)
`--offset <number>`	Number of projects to skip, for pagination

Projects are listed newest first with id, name, state, run and flow counts. --json returns { provider, meta: { offset, count, total }, projects: [...] } with a dashboard url per project.

Espresso

Run Android Espresso tests on real devices and emulators.

testingbot espresso [appFile] [testAppFile] [options]

Arguments:

appFile - Path to application APK file
testAppFile - Path to test APK file containing Espresso tests

Device Options:

Option	Description
`--app <path>`	Path to application APK file
`--test-app <path>`	Path to test APK file
`--device <name>`	Device name (e.g., "Pixel 6", "Samsung.*")
`--platform-version <version>`	Android OS version (e.g., "12", "13", "14")
`--real-device`	Use a real device instead of an emulator
`--tablet-only`	Only allocate tablet devices
`--phone-only`	Only allocate phone devices
`--locale <locale>`	Device locale (e.g., "en_US", "de_DE")
`--timezone <timezone>`	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
`--name <name>`	Test name for dashboard identification
`--build <build>`	Build identifier for grouping test runs
`--test-runner <runner>`	Custom test instrumentation runner
`--language <lang>`	App language (ISO 639-1 code, e.g., "en", "fr", "de")

Test Filtering:

Option	Description
`--class <classes>`	Run tests in specific classes (comma-separated fully qualified names)
`--not-class <classes>`	Exclude tests in specific classes
`--package <packages>`	Run tests in specific packages (comma-separated)
`--not-package <packages>`	Exclude tests in specific packages
`--annotation <annotations>`	Run tests with specific annotations (comma-separated)
`--not-annotation <annotations>`	Exclude tests with specific annotations
`--size <sizes>`	Run tests by size: small, medium, large (comma-separated)

Network & Location:

Option	Description
`--throttle-network <speed>`	Network throttling: 4G, 3G, Edge, or airplane
`--geo-location <code>`	Geographic IP location (ISO country code, e.g., "US", "DE")

Tunnel:

Option	Description
`-t, --tunnel`	Start a TestingBot tunnel for this test run (cannot be combined with `--async`)
`--tunnel-identifier <id>`	Identifier for the tunnel, allowing multiple tunnels in parallel

Output Options:

Option	Description
`--async`	Start tests and exit without waiting for results
`-q, --quiet`	Suppress progress output
`--json`	Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail
`--json-file`	Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file
`--json-file-name <path>`	Custom path for the JSON results file (requires `--json-file`)
`--report <format>`	Download report after completion: html or junit
`--report-output-dir <path>`	Directory to save reports (required with --report)

Examples:

# Basic usage with positional arguments
testingbot espresso app.apk app-test.apk --device "Pixel 8"

# Using named options
testingbot espresso --app app.apk --test-app app-test.apk --device "Pixel 8"

# Real device with specific Android version
testingbot espresso app.apk app-test.apk \
  --device "Samsung Galaxy S24" \
  --platform-version "14" \
  --real-device

# Run specific test classes
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --class "com.example.LoginTest,com.example.HomeTest"

# Run tests with annotations
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --annotation "com.example.SmokeTest" \
  --size "small,medium"

# With network throttling and geolocation
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --throttle-network "3G" \
  --geo-location "DE" \
  --language "de"

# Download JUnit report
testingbot espresso app.apk app-test.apk \
  --device "Pixel 8" \
  --report junit \
  --report-output-dir ./reports

XCUITest

Run iOS XCUITest tests on real devices and simulators.

testingbot xcuitest [appFile] [testAppFile] [options]

Arguments:

appFile - Path to application IPA file
testAppFile - Path to test ZIP file containing XCUITests

Device Options:

Option	Description
`--app <path>`	Path to application IPA file
`--test-app <path>`	Path to test ZIP file
`--device <name>`	Device name (e.g., "iPhone 15", "iPad.*")
`--platform-version <version>`	iOS version (e.g., "17.0", "18.2")
`--real-device`	Use a real device instead of a simulator
`--tablet-only`	Only allocate tablet devices
`--phone-only`	Only allocate phone devices
`--orientation <orientation>`	Screen orientation: PORTRAIT or LANDSCAPE
`--locale <locale>`	Device locale (e.g., "DE", "US")
`--timezone <timezone>`	Timezone (e.g., "America/New_York", "Europe/London")

Test Configuration:

Option	Description
`--name <name>`	Test name for dashboard identification
`--build <build>`	Build identifier for grouping test runs
`--language <lang>`	App language (ISO 639-1 code, e.g., "en", "fr", "de")

Network & Location:

Option	Description
`--throttle-network <speed>`	Network throttling: 4G, 3G, Edge, or airplane
`--geo-location <code>`	Geographic IP location (ISO country code, e.g., "US", "DE")

Tunnel:

Option	Description
`-t, --tunnel`	Start a TestingBot tunnel for this test run (cannot be combined with `--async`)
`--tunnel-identifier <id>`	Identifier for the tunnel, allowing multiple tunnels in parallel

Output Options:

Option	Description
`--async`	Start tests and exit without waiting for results
`-q, --quiet`	Suppress progress output
`--json`	Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail
`--json-file`	Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file
`--json-file-name <path>`	Custom path for the JSON results file (requires `--json-file`)
`--report <format>`	Download report after completion: html or junit
`--report-output-dir <path>`	Directory to save reports (required with --report)

Examples:

# Basic usage with positional arguments
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16"

# Using named options
testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16"

# Real device with specific iOS version
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 15 Pro" \
  --platform-version "17.2" \
  --real-device

# iPad in landscape mode
testingbot xcuitest app.ipa app-test.zip \
  --device "iPad Pro" \
  --tablet-only \
  --orientation LANDSCAPE

# With localization settings
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --locale "DE" \
  --language "de" \
  --timezone "Europe/Berlin"

# With network throttling and geolocation
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --throttle-network "3G" \
  --geo-location "DE"

# Download HTML report
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --report html \
  --report-output-dir ./reports

# Run in background
testingbot xcuitest app.ipa app-test.zip \
  --device "iPhone 16" \
  --async

Common Features

Real-time Progress

By default, the CLI shows real-time progress updates including:

Test status updates with actual device names (even when using wildcards)
Device allocation status
Live output from Maestro flows

Use --quiet to suppress progress output.

Graceful Shutdown

Press Ctrl+C to gracefully stop running tests. The CLI will:

Stop all active test runs on TestingBot
Clean up resources
Exit with appropriate status code

Press Ctrl+C twice to force exit immediately.

Report Downloads

All test frameworks support downloading reports after completion:

# JUnit XML format (for CI integration)
--report junit --report-output-dir ./reports

# HTML format (for human viewing)
--report html --report-output-dir ./reports

Artifact Downloads (Maestro only)

Download all test artifacts including logs, screenshots, and video recordings:

testingbot maestro app.apk ./flows --download-artifacts --build "my-build"

Artifacts are saved as a zip file named after the --build value (or with a timestamp if not provided).

Exit Codes

Code	Meaning
`0`	All tests passed (also for `--async`, `--dry-run`, and failed tests with `--json-file`)
`1`	CLI or infrastructure error: invalid arguments, missing credentials, upload failure, timeout
`2`	One or more tests failed

Distinguishing 1 from 2 lets CI decide whether to retry the job or fail the build.

JSON Output

--json prints one JSON document on stdout and moves all log lines to stderr, so testingbot maestro app.apk ./flows --json | jq works. --json-file writes the same document to disk while keeping the normal console output. Both flags imply --quiet.

{
  "provider": "maestro",
  "outcome": "failed",
  "success": false,
  "appId": 1234,
  "url": "https://testingbot.com/members/maestro/1234",
  "runs": [
    {
      "id": 5678,
      "status": "DONE",
      "passed": false,
      "device": { "name": "Pixel 6", "platform": "Android", "version": "14" },
      "url": "https://testingbot.com/members/maestro/1234/runs/5678",
      "flows": [
        {
          "id": 1,
          "runId": 5678,
          "name": "login",
          "status": "DONE",
          "passed": true,
          "attempt": 1,
          "latest": true,
          "startedAt": "2026-01-01T00:00:00Z",
          "completedAt": "2026-01-01T00:00:30Z",
          "durationSeconds": 30,
          "errors": []
        }
      ]
    }
  ]
}

outcome is one of passed, failed, started (--async), dry-run, or error. On error the document carries an error message and the exit code is 1.
flows (Maestro only) lists every attempt, including --retry re-runs. attempt counts from 1; latest marks the attempt whose verdict counts for the run.
runs is empty for --async, --dry-run, and errors raised before tests were submitted.

Documentation

For more information, visit TestingBot Documentation.

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
demo		demo
resources		resources
src		src
tests		tests
.gitignore		.gitignore
.npmignore		.npmignore
.nvmrc		.nvmrc
.prettierignore		.prettierignore
.prettierrc		.prettierrc
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
eslint.config.js		eslint.config.js
jest.config.js		jest.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Navigation Menu

Uh oh!

Repository files navigation

Installation

Authentication

Browser Login (Recommended)

Other Methods

Commands

Maestro

Organizing flows and subflows

Upload once, run many times

Status, artifacts and list

Espresso

XCUITest

Common Features

Real-time Progress

Graceful Shutdown

Report Downloads

Artifact Downloads (Maestro only)

Exit Codes

JSON Output

Documentation

License

About

Releases

Packages

Used by

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Installation

Authentication

Browser Login (Recommended)

Other Methods

Commands

Maestro

Organizing flows and subflows

Upload once, run many times

Status, artifacts and list

Espresso

XCUITest

Common Features

Real-time Progress

Graceful Shutdown

Report Downloads

Artifact Downloads (Maestro only)

Exit Codes

JSON Output

Documentation

License

About

Topics

Resources

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages