Firebase Test Lab: Automated App Testing on Real Devices in Cloud

In this tutorial, you will learn about Firebase Test Lab: Automated App Testing on Real Devices in Cloud. We cover key concepts, practical examples, and best practices to help you master this topic.

Firebase Test Lab runs your Android and iOS app tests on real and virtual devices hosted in Google's data centers, catching compatibility and performance issues before release.

What You'll Learn

How to use Firebase Test Lab to run instrumentation tests, Robo automated tests, game loops, configure device matrices, and analyze test results including screenshots and logs.

Why It Matters

Testing on every device variant is impractical. Test Lab runs your tests on hundreds of device/OS combinations in parallel. DodaTech runs its Android antivirus app against 50+ device configurations before every release.

Real-World Use

Before releasing a new scan algorithm, the QA team uploads the APK to Test Lab, selects 20 popular Android devices, runs Robo tests on each, and reviews screenshots and crash logs — all in under 30 minutes.

flowchart LR
    A["Upload APK /\nIPA + Test Suite"] --> B["Test Lab\nCloud"]
    B --> C["Device Matrix\nRun in Parallel"]
    C --> D["Device 1\nGalaxy S24"]
    C --> E["Device 2\nPixel 8"]
    C --> F["Device 10\niPhone 15"]
    D --> G["Results\nScreenshots + Logs"]
    E --> G
    F --> G
    G --> H["Pass /\nFail Report"]
    style A fill:#dbeafe,stroke:#2563eb
    style B fill:#fef3c7,stroke:#d97706
    style G fill:#bbf7d0,stroke:#16a34a

Running a Robo Test

# Install Firebase CLI
npm install -g firebase-tools

# Run a Robo test (automated crawl without writing tests)
firebase test:android run \
  --type robo \
  --app app-debug.apk \
  --device model=Pixel8,version=34 \
  --device model=GalaxyS24,version=34 \
  --device model=Pixel7,version=33 \
  --timeout 5m

# Expected output:
# Starting Robo test...
# Device 1 (Pixel8): Running...
# Device 2 (GalaxyS24): Running...
# Device 3 (Pixel7): Running...
# All devices completed.
# Results: https://console.firebase.google.com/testlab/...

Running Instrumentation Tests

# Run Espresso / UI Automator tests
firebase test:android run \
  --type instrumentation \
  --app app-release.apk \
  --test app-test.apk \
  --device model=Pixel8,version=34,locale=en,orientation=portrait \
  --device model=GalaxyS24,version=34 \
  --device model=iPhone15,version=17 \
  --timeout 30m

# Expected output:
# Starting instrumentation test...
# Pixel8: 12/12 tests passed
# GalaxyS24: 12/12 tests passed
# iPhone15: 10/12 tests passed (2 failures)
# ⚠  See detailed results in Firebase Console

Configuring Device Matrix

# Define device matrix in a YAML file for reuse
# matrix.yaml
# devices:
#   - model: Pixel8
#     version: 34
#     locale: en
#     orientation: portrait
#   - model: GalaxyS24
#     version: 34
#     locale: en
#   - model: Pixel7
#     version: 33
#   - model: Pixel6a
#     version: 32
#   - model: OnePlus12
#     version: 34

firebase test:android run \
  --type robo \
  --app app-release.apk \
  --device-matrix matrix.yaml \
  --timeout 10m

iOS Testing

# Run iOS tests via Test Lab (requires XCTest)
firebase test:ios run \
  --type xctest \
  --test-ipa ScanUITests.ipa \
  --device model=iphone15pro,version=17 \
  --device model=iphone14,version=16 \
  --timeout 30m

# Expected output:
# iPhone 15 Pro (17.0): All tests passed
# iPhone 14 (16.0): All tests passed

Analyzing Results

// Access test results via Firebase Admin SDK
const admin = require("firebase-admin");

async function getTestResults() {
  const results = await admin.testLab().listTestMatrices({
    filter: "create_time > '2026-06-27'"
  });

  results.forEach((matrix) => {
    console.log("Test Matrix:", matrix.testMatrixId);
    console.log("State:", matrix.state);
    console.log("Outcome:", matrix.outcomeSummary);

    if (matrix.outcomeSummary === "FAILURE") {
      console.log("Failed devices:");
      matrix.testExecutions.forEach((execution) => {
        if (execution.state !== "VALID") {
          console.log(" -", execution.environment.androidDevice.model);
        }
      });
    }
  });
}
// Expected output: Test Matrix: matrix_abc123
//                  State: FINISHED
//                  Outcome: PASS

Common Mistakes

1. Using Debug Builds for Testing

Debug builds behave differently from release builds. Run tests against release builds to catch issues that only appear in production (ProGuard/R8 issues, missing permissions).

2. Not Setting Test Timeout

Without a reasonable timeout, tests can run indefinitely. Set --timeout to Fail Fast and save costs.

3. Too Many Device Variants

Testing on 100+ devices in a single run takes hours and costs more. Select the top 10-15 most popular devices for your target market.

4. Ignoring Robo Test Limitations

Robo tests only exercise the UI automatically. They won't test specific business logic, API integrations, or scenarios requiring specific inputs.

5. Not Reviewing Screenshots

Test Lab captures screenshots during Robo tests. Review these to catch visual issues (truncated text, overlapping elements, wrong colors).

Practice Questions

1. What is the difference between Robo and instrumentation tests?
2. How do you configure a device matrix for regional testing?
3. How does Test Lab handle test failures?
4. How do you save test results for Compliance?

Answers:

1. Robo tests automatically crawl the app without test code. Instrumentation tests run your custom test code (Espresso, XCTest) for specific scenarios.
2. Specify device models, OS versions, locales, and orientations in the device matrix. Include devices popular in your target region.
3. Test Lab reports which devices pass/fail, provides logs and screenshots per device, and aggregates results in the Console.
4. Test Lab stores results including logs, screenshots, and performance data. Export results to Cloud Storage for long-term retention.

Challenge: Create a Test Lab test suite for Durga Antivirus Pro: configure a device matrix with 10 top Android devices, write a Robo test for the scan flow, run instrumentation tests for threat detection, and review the results dashboard.

FAQ

Mini Project

Set up Test Lab for a security app: create a device matrix of 5 Android + 3 iOS devices, run a Robo test on the debug build, review screenshots for UI issues, fix any detected problems, and add instrumentation tests for the scan flow.

What's Next

Firebase Project End-to-End — build a complete project combining all Firebase services.