Cloud & DevOps Services

Three environments, and why two is not enough

A surprising number of live business systems have exactly one environment: production. Changes are tested by deploying them to real users, and the rollback plan is the developer's memory.

We set up three. Local, so developers can work without touching anything shared. Staging, which mirrors production as closely as budget allows and holds a sanitised copy of real data — the environment where you review a change before it becomes everyone's problem, and where database restores are tested. Production, which nobody edits by hand. The staging environment is the one clients question because it costs a little more per month. It is also the one that pays for itself the first time a migration behaves differently against real data volumes than it did against ten test rows.

What a deployment pipeline actually does

"CI/CD" is a vague term for a concrete thing: a script, triggered by a push, that does the same steps in the same order every time so nobody forgets one at 11pm.

Ours typically runs: install dependencies, run the test suite, run a linter and a dependency vulnerability scan, build assets, then on the main branch deploy to staging automatically. Production deployment is one manual approval away — not automatic, because someone should decide when customers get a change. The deployment itself puts the new release in place, runs migrations, warms caches and switches over, so users are not served a half-updated application. And it can be reversed: a rollback path we have actually exercised, not one we assume works.

The test that matters is simple. Can you deploy twice on a Friday afternoon without anxiety? If not, the pipeline is not finished.

Backups: the only ones that count are the ones you have restored

This is the part of infrastructure work we are least willing to compromise on, because it is the difference between a bad afternoon and the end of a business.

Three requirements. Backups must be automated, because a manual backup is a backup that stops happening in week three. They must be off-server and ideally off-provider — a backup on the same machine as the data protects you from your own mistakes and nothing else, and a backup in the same account as the thing it protects shares a single point of failure. And they must be tested: a backup nobody has restored is a hypothesis, not a safeguard. We restore one into staging on a schedule and confirm the data is complete.

Alongside that we agree two numbers with you explicitly, because they drive the design and the cost: how much data you can afford to lose, and how quickly you must be back up. Most small businesses answer "a day" and "a few hours", which is inexpensive to deliver. Answering "nothing" and "minutes" is also achievable and costs considerably more — the point is to decide deliberately rather than discover it during an incident.

Secrets, and the credential nobody rotated

Database passwords, API keys and payment credentials belong in environment variables or a secret manager — never in the repository, never in a shared spreadsheet, never pasted into a chat thread. If a secret has ever been committed to Git it is compromised permanently, because it remains in the history, and it must be rotated rather than deleted.

We also separate credentials per environment, so a leak from staging cannot touch production, and we keep a documented list of every credential the system depends on, who owns it, and where it lives. That list is what makes it possible to rotate anything at all when a laptop is lost or a contractor's access ends.

Monitoring that tells you something before your customers do

Uptime checks that only ping the homepage miss the most common real failure: the server is up and returning errors to everyone. So we monitor uptime from outside your network, error rate and response time in the application, queue depth so stalled background jobs are visible, disk and memory with alerts well before exhaustion, certificate expiry, and backup success or failure as an explicit signal rather than an assumption.

Two rules about alerts, learned the hard way. They must reach a person — a phone, not an unread mailbox. And there must be few enough of them to be believed; a channel that fires forty times a day trains everyone to ignore the one that mattered. Tuning alerts down is real work and worth doing.

Cost control, before the surprise invoice

Cloud bills grow quietly. The recurring causes we find: environments left running after a project ended, snapshots and backups accumulating with no retention policy, over-provisioned instances sized for a launch spike that never returns, logs retained for years by default, and data-transfer or NAT charges nobody predicted — usually the first genuine surprise on an AWS invoice.

So we set billing alerts on day one, apply retention policies to logs and snapshots, tag resources so spend can be attributed, and right-size after a month of real traffic rather than guessing up front. For choosing a tier in the first place — and for the honest case that a VPS beats a major cloud for most small businesses — see our hosting comparison.

Writing down what to do at 2am

Incidents are handled badly not because people lack skill but because they are improvising under pressure. A short runbook fixes most of that: how to check whether the problem is the application, the database or the network; how to put the site into maintenance mode; how to roll back the last release; how to restore the database and how long it takes; who to call at the hosting provider and what account details they will ask for; and who on your side must be told.

Two pages, written calmly in advance. We produce it as part of setup and keep it somewhere reachable that is not the server that just went down — a detail that sounds obvious and is routinely got wrong.