Keeping a Jenkins DevOps Estate Maintainable
Author : James Smith | Published On : 21 Oct 2026
Our Jenkins DevOps installation had four hundred jobs, eleven plugins nobody could account for, and a configuration built over six years by people who had mostly left.
It worked. Nobody could change it safely. Here is how we got it back under control.
Configuration in a user interface does not scale
The root problem.
Jobs created by clicking are not reviewable, not reproducible, and not recoverable. When our controller failed, restoring it meant restoring a backup and hoping the backup was current.
Everything is now defined in code, in version control, reviewed like any other change. The controller holds no configuration that is not reproducible from source.
A Jenkins DevOps estate configured through the interface will drift until nobody dares touch it.
Audit the plugins
The most common source of fragility.
We had eleven plugins installed, four of them unmaintained, two duplicating each other, and one that nobody could explain. Each was a dependency on an upgrade path.
We now keep a declared plugin list, reviewed quarterly, with anything unmaintained scheduled for replacement. Upgrades became routine rather than a project once that list was short.
Build in containers, not on the controller
The change that removed most of our environment problems.
Jobs used to run on long lived agents configured by hand, which drifted. A build working on one agent would fail on another, and the cause was always something installed years earlier.
Builds now run in containers holding exactly the toolchain that project needs. The agent provides compute and nothing else.
Keep the controller doing one thing
Controllers accumulate responsibilities.
Ours was running builds, storing artefacts, hosting a reporting dashboard, and serving as a file drop for three teams. Each addition was reasonable and together they made restarting it a negotiation.
The controller now schedules work. Artefacts live in a registry, reports live in the reporting platform, and nobody uses it as a file server.
Keep the agents disposable
Long lived agents accumulate state.
Ours are created from an image, used, and destroyed. Nothing persists between builds except deliberate caches held outside the agent.
Before that, agents drifted. A build succeeding on one and failing on another was our most common support question, and the cause was always something installed years earlier by somebody who had left.
Treat the pipeline as the product's code
Same repository, same review, same standards.
A pipeline change is a change to how the product is built, tested and released. It deserves the same scrutiny as anything else and frequently receives none.
Our pipeline definitions live alongside the application, and a change to either goes through the same review.
Measure the pipeline
Four numbers, reported weekly.
How long a build takes. How often it fails for reasons unrelated to the code. How long from commit to production. And how often a pipeline failure blocks a team.
That second one is the important one. A pipeline failing for its own reasons teaches people to re-run rather than investigate, which is how a real failure gets ignored.
Those measures feed our wider devops KPIs rather than sitting in a separate report, because pipeline health is delivery health.
Decide what runs where
Controllers get asked to run everything.
Ours now runs continuous integration and deployment orchestration. Scheduled data jobs moved to the platform's own scheduler, long running analytical work moved to dedicated compute, and three teams' ad hoc scripts moved into their own repositories with their own pipelines.
Each of those had arrived because the controller was convenient. Together they made it a single point of failure for work that had nothing to do with building software.
A Jenkins DevOps controller doing six jobs is six reasons it cannot be restarted during the working day.
Plan the upgrades
Falling behind is a security position.
We upgrade the controller quarterly, through a test instance first, with the plugin list reviewed each time. The short plugin list is what makes that possible.
Before we trimmed it, we were eighteen months behind and each upgrade attempt broke something nobody could predict.
Keep a second person who understands it
The risk nobody logs.
Our estate was understood by one engineer for about four years. He turned up every day, so nobody treated it as a single point of failure.
Two people now know every significant part, with deliberate rotation. The cost is a few days a quarter and it has already saved us once.
Connect it to the systems around it
Pipelines do not exist alone.
Ours deploys to several environments, updates ticket status, and notifies the teams who need to know. One pipeline also triggers a configuration deployment handled by our Salesforce Implementation Partner, because that platform has its own release process and its own constraints.
Those connections need owners and monitoring. An integration that fails silently means a release that looks complete and is not, which our Salesforce Implementation Partner found the hard way once.
What our standard requires
- All configuration in code, reviewed, reproducible.
- A short declared plugin list, reviewed quarterly.
- Builds in containers with pinned toolchains.
- A controller that only schedules work.
- Pipeline definitions alongside the application code.
- Four pipeline measures reported weekly.
- Quarterly upgrades through a test instance.
Common questions
Is Jenkins still the right choice?
For us, yes. It is flexible, we understand it, and the cost of moving four hundred jobs would exceed the benefit.
Who should own the estate?
A named team with allocated time. A Jenkins DevOps installation owned by everybody drifts until nobody will touch it.
How many plugins is reasonable?
As few as possible. We run five and each has a stated reason.
What causes most pipeline failures?
Environment drift, before we containerised builds. Now it is mostly genuine test failures, which is the correct answer.
Where should a team start?
Configuration in code. Everything else about a Jenkins DevOps estate becomes possible once the controller is reproducible.
Where we are now
Four hundred jobs defined in code, five plugins, builds in containers, and quarterly upgrades that nobody dreads.
The installation does roughly what it always did. What changed is that it can be rebuilt from source, upgraded on schedule, and modified by somebody who was not there when it was built.
