A while back, a number of brands were being divested from their parent company. The internal IT team that ran their websites were also being disbanded, so we were asked if we could take these on.
We could. After all, we’d built the websites. What could possibly go wrong?
What then followed was six months of making no money on the deal. Why? Our costing was off because when we run websites and host them for clients over at Standfirst, we pay the hosting bills. And that means we monitor, track and watch our costs. We’d succeeded in taking plenty of client sites down from running on, for example, four 32GB servers and getting them happily running on a cloud platform VPS of 1GB – albeit with some services running elsewhere, such as Redis.
As a consequence, we looked at the sizes of the sites, and the traffic, did some sums, and worked out we could charge around £2k a month (these are big, popular sites with many many users) and clear £1k a month of profit.
Our missing profit
In reality, we cleared £0 a month in profit. £0 profit is sub-optimal and a poor way to run a business. A couple of the smaller sites did clear a reasonable profit, but the big ones did not.
I was discomfited by this. We built those websites. They should be efficient, like the ones we’d built and hosted for many other customers.
Except… there turned out to be a big difference in the websites we built and hosted, and the websites we built and this client hosted. We weren’t responsible for the bills, we didn’t have server level monitoring, and we had to do everything at arm’s length. So over the years costs crept up yet we had no idea. We weren’t in those conversations. Our job was to solve client problems, and the clients weren’t presenting this as a problem, and we had no visibility to know it was a problem.
These weren’t generally because the code was bad, but simply because the differences between our local and the production configurations meant that things simply didn’t show up for our team in our testing. We didn’t see the interactions between the apps, the CDNs and the object caching tools that ran on the servers.
Digging deep for big savings
So when we took over the hosting of these website our initial thought was “hmmm, this isn’t great.” We quickly got everything reliable on the new stack, and then soon got to work on reducing the amount of auto-scaling. None of this was charged directly to the client – after all, it was for our bottom line, not theirs. If we could halve our server costs we’d make £30k a year more money, and this would be every year. But what our efficiency drive did help with was in not having to have that awkward conversation with the client about increasing the cost of their hosting. Something we’d estimated for our own code.
£30k a year in savings is some motivation!
By having our developers and devlops in the same team we removed the siloes, we reduced costs and we reduced the environmental consequences as well – fewer, smaller machines always use less energy and generate less CO2 as a result. We also now have a far better understanding of how to host and manage large websites, make them faster, and be economical about it.
The result of having a financial imperative was that our motivations became focused on an end result – lower costs. And reliability was maintained because we were also the team that would be called out at night – and we don’t pay callout bonuses!
How could you make that work in your company, or with your supplier?
I believe the following strategies work:
Zero based budgeting – each year or contract period, the supplier or department has to create a budget and account for everything. But you give some wiggle room in case they make errors and to prevent aggressive short term cost-cutting that damages the team, and you define some prioritised goals for ongoing expenses and have a budget for that. e.g. they may need £10k extra this year to spend on tuning, but that will save £20k the next year. If the £20k saving comes true, you provide a bonus at the end of the year of half of the saving.
Responsibility and reward handover – my thesis I set out with here is that you hand responsibility for certain costs over to the people who can most influence those costs. If you’re trying to save fuel on a fleet of lorries, give fuel economy responsibilities to the driver’s team and give them the motivation to adjust driving styles. If you’re trying to save on repair costs for laptops, work out how much it’s costing you to repair them all, divide it by the employees, and everyone who doesn’t need a repair gets a small extra bonus.
Profit sharing – profit sharing incentives can help incentivise employees, but what about doing it with suppliers? Savings of £x could result in a bonus to a supplier based on the formulas below. It always amazes me that, as a supplier, nobody has ever considered giving us any incentives other than the chancers who seem to think we’ll build their idea for free in return for some equity. But why not? This can work especially well for partnerships, and it does happen in other industries.
In all cases, this takes maths and effort. But the equations are usually pretty simple. You need what’s called a vesting decay model coupled with a traffic adjusted metric. That means there’s an ongoing bonus for the savings over a number of years.
So let’s say we take that £60k hosting bill:
In Year 1, the team optimises the stack, dropping the run rate to £40,000 (Saving £20,000).
In Year 2, they maintain that efficiency but don’t find any new savings.
In Year 3, they find a new optimisation, dropping the run rate to £30,000 (Saving an additional £10,000).
In Years 4 and 5, they maintain the £30,000 run rate.
| Timeline | Actual Stack Cost | New Savings Achieved | Maintained Savings Payout | Total Team Bonus Pool | Net Company Savings (Retained) |
| Year 0 | £60,000 | — | — | £0 | — |
| Year 1 | £40,000 | £20,000 (Yr 1) | — | £10,000 (50% of £20k) | £10,000 |
| Year 2 | £40,000 | £0 | £20,000 (Yr 2) | £5,000 (25% of £20k) | £15,000 |
| Year 3 | £30,000 | £10,000 (Yr 1) | £20,000 (Yr 3) | £7,000 (50% of £10k + 10% of £20k) | £23,000 |
| Year 4 | £30,000 | £0 | £10,000 (Yr 2) | £2,500 (25% of £10k) | £27,500 |
| Year 5 | £30,000 | £0 | £10,000 (Yr 3) | £1,000 (10% of £10k) | £29,000 |
| Year 6+ | £30,000 | £0 | — | £0 | £30,000 (Forever) |
This prevents a bonus cliff, so it stops developers wanting to wait on releasing optimisations. At the same time it provides on ongoing saving and permanent long-term reductions in overheads.
To help, I’ve created a spreadsheet to help create some possible bonus structures you could use, which can also be visited directly in Google Docs:
Hopefully, with this approach, developers won’t treat cloud infrastructure as an infinite resource, money can be saved, and environmental impacts reduced.
