How do you keep your software reliable?

"How do you keep your software reliable?"

Software reliability

Ever wondered how we, at 1st Money, keep our software so reliable? In short, we do it by every day:

• Exactly matching all 10,000 of HMRC's model tax calculations,
• Publishing 280 product videos,
• Running over 6,000 click-navs,
• Checking over 34,000 asserts,
• And running over 1,000 delves.

The video's numbers are up-to-date as of the 22nd of April, 2024. Keep watching, to see how it all works.

Tax scenarios

We test our tax calculations by inventing 15 different imaginary taxpayers, shown (at the top of the screen).

Each one has different settings. For example, how many hours the imaginary taxpayer works, how much is paid, whether the imaginary taxpayer's married, whether there's children, and much more. We use our 15 different imaginary taxpayers to exactly match all 10,000 of HMRC's model tax calculations.

Each tax calculation is matched to the penny, giving us recognition from HMRC that our tax software is 100% correct. And, of course, we sit down and re-verify them all with HMRC every year.

Product videos

So far, we've made 280 tutorial videos, explaining every part of the 1st Money product.

Everywhere in the 1st Money product, you'll find video buttons . When you tap one, a video pops out from the side. In our help pages, almost every sentence can be tapped on to show a video explaining that step.

To make it all work perfectly, we've built our own video player, and video creation software. The videos mean that any customer that wants help can immediately find answers.

Questions and answers

We've written 2,600 questions and answers, explaining even more parts of the 1st Money product.

The questions and answers are searchable through an AI powered search box. When you type a question, the AI interprets your question's meaning, and then shows you the best matching answers.

It's fast, and it's accurate, and we're always adding more answers.

Elements

To make our software look beautiful on both laptops and smartphones, we've built our own library of 40 software widgets. Each widget has its own testing page, showing the different ways it can be used. The testing pages help us spot if any of the elements has any errors.

Click tests

Every night, we use robots to do 10,000 taps through the 1st Money product on 6,000 pathways. The taps push our test server's 8-core CPU to 100% for over 5 hours straight.

On each pathway, a tiny robot taps a different path through the 1st Money product as fast as the CPU allows. It's like you're seeing on-screen right now. If the robot spots anything wrong, it takes some screenshots and writes some notes about the error. Using the error notes, our software engineers then make fixes.

By using click tests, bugs are usually spotted by us first, and not by our customers.

Engineering tools

At 1st Money, all our software engineers use specialised engineering tools for their day-to-day programming.

Every time a software engineer finishes making changes to the software, the software engineer tools analyse the code the software engineer's written. The analysis warns the software engineer about errors like missing functions, unused variables, and much more.

TypeScript

Every software engineer, twice a day, runs a script that compiles all 200 of our TypeScript files, and then scans them with ESLint. We use TypeScript to make sure any JavaScript errors get caught by us first, and not by our customers.

Unit tests

Every software engineer, twice a day, runs thousands of asserts to test our software. The asserts include 10,000 tax asserts, and 21,000 system asserts.

Each dot you can see on the screen represents about twenty asserts. Altogether, the 34,000 asserts take about 1 minute to run. The asserts make sure that day-to-day software changes don't accidentally break our existing code.

Delves

Every software engineer, twice a day, runs 1,000 code checks, taking about 2 minutes for the entire set. The code checks scan the 6,000 files in our codebase, looking for things to improve in the code, including:

• Code that's too complicated,
• Things that need to be named more clearly,
• And much more.

The delves mean that each day's changes are continuously scrubbed clean.

Confirm URLs

Every software engineer, twice a day, runs a spider that automatically opens and tests a sample of over 500 pages, spread across 1st Money's entire system. It takes about 1 minute each time, to test them all.

The spider helps make sure that obvious problems like dead links or page-load errors are caught by us first, and not by our customers.

Documentation quality

Every software engineer, twice a day, runs a script that scans all our documentation and help, searching for sentences that can be improved. We check for things like:

• Spelling,
• Abbreviations,
• Sentences that're too long,
• And much more.

We hope you love it!

And that's it! That's everything you need to know about how we keep our software reliable at 1st Money. We've loved building it, and hope you love it too!