A Mobile App Testing Services Suite People Trust
Author : James Smith | Published On : 04 Nov 2026
Our automated mobile suite had a seventeen per cent intermittent failure rate. Engineers reran it as a reflex. A real failure was indistinguishable from the usual noise.
We spent four months fixing that before adding a single new test. Here is what Mobile App Testing Services look like once the suite is trustworthy.
A flaky suite has negative value
The honest accounting.
It consumes compute, it consumes engineer attention, and it produces a signal nobody believes. Worse, it trains people to ignore failures, which is exactly the habit that lets a real defect through.
We had two genuine failures masked by noise in one quarter. Both reached users, and both had been caught by the suite and dismissed.
Most flakiness is waiting for the wrong thing
The single largest cause in our case.
Tests that wait a fixed number of seconds pass on a fast device and fail on a slow one, or pass on a quiet network and fail on a busy one.
We replaced every fixed delay with a wait for an observable condition. That change alone took us from seventeen per cent to about five, and it was mechanical work rather than clever work.
Control the state the test starts from
The second largest cause.
Tests that depend on data left by a previous test fail whenever the order changes or one of them is skipped.
Each of ours now establishes its own starting state through a programmatic route rather than through the interface, and tears it down afterwards. Slower per test and dramatically more reliable.
Stop testing through the interface where you do not need to
A scope decision, not a technique.
Logging in through the interface to test a settings screen is testing the login forty times a run for no reason.
Setup happens programmatically. The interface is exercised only where the interface is what is being tested, which removed about a third of our interface interactions and a corresponding share of the fragility.
Quarantine rather than disable
The policy that prevented silent erosion.
A test failing intermittently moves to a quarantine set. It still runs, it does not block anything, and it appears on a list with an owner and a date.
Disabling tests makes the suite green and the coverage fictional. Our quarantine has never exceeded eleven tests and nothing has stayed in it longer than three weeks.
Make the failure tell you what happened
Debuggability as a design requirement.
Every failure captures a screenshot, the view hierarchy, the device logs, and the network activity of the run.
Before this, a failure produced an assertion message and an engineer spent twenty minutes reproducing it. Now most failures are diagnosed from the artefacts without rerunning anything.
Run the fast set on every change
The only way it stays healthy.
Our nine minute set on four devices gates every merge. A suite that runs nightly accumulates failures in batches and nobody knows which change caused what.
Mobile App Testing Services that cannot gate a merge are reporting history rather than preventing defects.
Know the state of every device
A cause of flakiness that is not in the code.
A handset low on storage, on an unexpected operating system version, with a stale build installed, or thermally throttled will produce failures that look like test problems.
Our Asset and Configuration Management over the lab tracks version, storage, battery health, and installed builds for every device. Roughly one in eight of our remaining intermittent failures turned out to be a device condition rather than a test.
Assess automated test generation honestly
Because the claims are large.
We evaluated a multi-agent AI architecture that proposed test cases, critiqued them against the requirement, and refined them before output.
The generated cases were well formed and mostly duplicated existing coverage. The genuinely novel ones were about one in five, and two of them asserted behaviour the requirement did not specify. It is a useful prompt for an exploratory session and it is not a substitute for somebody who understands the product.
Measure the suite, not just the software
Four numbers reviewed weekly.
Intermittent failure rate. Mean time to diagnose a failure. Suite duration. Quarantine size and age.
Those four tell you whether the suite is an asset or a liability, and none of them is about the application under test.
What our suite requires
- No fixed delays anywhere, only waits on observable conditions.
- Each test establishes and tears down its own state programmatically.
- Interface interaction only where the interface is what is tested.
- Quarantine with an owner and a date, never disabling.
- Screenshot, view hierarchy, logs, and network captured on failure.
- A nine minute set gating every merge.
- Tracked device state across the entire lab.
- Weekly review of flakiness, diagnosis time, duration, and quarantine.
Common Questions
What flakiness rate is acceptable?
Near zero. Above about two per cent people start rerunning rather than investigating, and the suite stops functioning as a signal.
Should every test be automated?
No. Exploratory work finds things automation cannot, and Mobile App Testing Services that automate everything lose that entirely.
How long should a gating suite take?
Under ten minutes. Ours is nine on four devices, and the full set runs nightly.
Is test generation useful?
As an aid for exploratory sessions, modestly. Our trial produced genuinely novel cases about one time in five.
How long does this take?
Ours took four months with no new coverage added. Mobile App Testing Services teams under pressure to add tests will not get that time unless somebody argues for it.
What is the single highest return fix?
Removing fixed delays. That took us from seventeen per cent to about five.
Who should own the device state?
Whoever owns the lab, under proper Asset and Configuration Management. An unknown device state makes a failure unexplainable.
Does this apply to hosted device farms?
Equally. Mobile App Testing Services running on hosted devices still need to know the version and the installed build they ran on.
Why track device condition?
Because about an eighth of our residual flakiness was storage, thermal, or version state rather than anything in the tests.
Where We Are Now
Intermittent failures under one per cent, a nine minute gate, failures diagnosed from artefacts without reruns, and a quarantine that rarely exceeds a handful of tests.
We added almost no coverage in those four months. We made the existing coverage believable, after which people started reading the results again.
