Background

We inherited a Rails app that hadn't been meaningfully maintained in about five years. 
Ruby 3.0, Rails 6.1, Webpacker, Bootstrap, Turbolinks, jQuery-era JavaScript.
No tests (not even one), no CI, no linter, no static analysis, no error tracking.

It was also live, it handled money, and third parties had integrated against it years earlier and were still sending traffic.

We rebuilt it from scratch on a modern stack, and once parity was proven, added features on top.
This is what we learned.

#1 Setting a rebuilt policy

Two policies are set explicitly:

  1. On the interface, reproduced exactly, even where it's wrong.
    Since this is a reimplementation against a frozen contract, it is not possible to change what the third party integrators are still depnding on. Even if there are things a reviewer would flag on sight, that break every integrator the day you "fix" them.
  2. Everything behind the interface was ours. Fixed deliberately, behind the scene.

#2 Write the old app's spec before writing any new code

The first artifact wasn't code. It was two documents about the app we were replacing:

  • A behavioural spec - what the old app does today, flow by flow, purely descriptive. No opinions. Where the code did something surprising, it stated the behaviour plainly and left the judgement alone.
  • An improvement review - bugs, performance, security, data integrity, each finding numbered, with severity.


Keeping those separate is the part I'd insist on again. A single document mixing "here's what it does" with "here's what's wrong with it" is unusable during a rebuild, because you consult it in two completely different moods. 
When you're porting a flow you want the description and nothing else. When these two documents are merged, you can't tell whether a paragraph is describing behaviour or arguing about it.

#3 Plan the execution in phases, each landing with its own tests

We split the migration into phases:
0 - foundation and CI
1 - data layer
2 - outbound service clients
3 - inbound authentication and audit
4 - the flows themselves
5 - views and front end
6 - security hardening
7 - deployment and parity verification

Each phase lands with its tests, so parity is provable rather than assumed. That sounds like ordinary discipline, but in a rewrite it does something specific: it prevents the failure where you port everything at 80% and then spend the remaining time discovering it multiplies out to a system that doesn't work.

#4 Capture the old app's traffic before you switch it off

This was the highest-leverage decision of the project.

Before cutover we captured a corpus of real traffic from the old app in its test environment.

  • The request logs
  • The changes in the database

Then a harness that rebuilds each request from the captured bytes, by referencing the steps transcribed from the integration guide. Finally replays it against the new app, and diffs the result, field by field.

That inversion is the whole trick. Most parity tests assert that specific things match, which catches regressions in the fields you thought to list. This one fails on any unlisted difference, which means it also catches the improvement nobody declared. If left uncaught, they quietly changes what were written into logs or databases, even if it is a "cleaner" or "correct" behavior.

With this divergence list, we have a changelog of intentional behaviour changes.

Example of what we discover in our tests:

  1. A "" that became NULL
  2. Field ordering in json
  3. And even an audit row that had been silently lost for years

#5 Reading the old source tells you what the code says, not what shipped

Related lesson, different medium (#4 is log, #5 is screenshot).

We rebuilt the result screens from the old templates, without referring to screenshots. They drifted in many ways (wrong text colour and weight, missing section breaks, etc).

So we went back with screenshots and rebuilt against pixels, then verified by rendering both pages in a browser and reading every box's geometry out of the DOM against coordinates sampled from the original image. Everything landed within a fraction of a pixel.

The general form: the old codebase is a description of intent; the running system is the specification. Where they disagree, and they will, the running system wins, and you need captures - logs, screenshots, database rows, to know which is which.

#6 Porting is mostly deleting

The new app is bigger than the old one (about 3X more codes). 
But a surprising fraction of the work was removing things. For example:

a) A ten-second blocking sleep on a request thread, waiting for a result to appear, became a polling page.
b) Several AJAX routes were dead. They are removed and not ported.

Everything deleted got a line in the plan saying it was deleted and why. 
"We didn't port this" is exactly the kind of decision that looks like an oversight six months later.

#7 The security work paid for itself in a way we couldn't have planned

Midway through, an external security assessment landed against the live old app: a critical value-tampering finding. The value was rendered into the page as a data attribute, the browser read it back out and put it in the payload sent to upstream.

The rebuild had closed it in an earlier phase, as a matter of course: payloads are rebuilt server-side, and read from the browser.

This is not a hardening measure we schedule. It is an architectural property we get for free when we decided to implement it earlier.

#8 The system test "issue"

Our system test runs in headless Chrome. Every assertion passes, but about one run in three, the driver silently drops a synthesised click or keystroke. We tried a few different approach, before finally discover it could be a Chrome bug. A fix is still work in progress.

References:

  1. https://github.com/teamcapybara/capybara/issues/2800
  2. https://issues.chromium.org/issues/402796660

#9 We sized a rate limit without data

In the old app, no rate limit is implemented. We added rate limit in the replacement app during the security hardening phase. We use a flat per-minute limit, it blocked the opening seconds of a legitimate traffic spike. We replaced it with two tiers: a burst shaped like a real spike, and a sustained ceiling no legitimate caller reaches.

The result

The stack modernisation - new Rails, new Ruby, new asset pipeline, no Node toolchain, CSS framework swap. This was the easy part, and it was maybe a fifth of the work. The other four fifths was figuring out what the old thing actually did, proving the new one did the same, and being able to show our working afterward.