Midnight Ride: Production Errors That Fix Themselves (Almost)

I wrote this article with the help of AI.

Some of my Rails apps record their production errors with Solid Errors, which can email you about them. It emails on every occurrence, though, so a single error that kept happening sent me an email about once a minute and used up my Mailgun allowance. I turned the emails off, and then I heard about errors only when I remembered to look.

In addition, I didn’t want fixes to take much manual effort, or to have to wait for me. If I’m on the road and an error happens in production, I want to be notified in real time, on my phone, and I want the fix to be prepared for me. Now that coding agents are good enough, why code the fix by hand? Why not automate bug fixes?

So I built Midnight Ride, named after Paul Revere’s ride to warn that the British were coming. When one of my apps errors in production, it tells me on Telegram and starts a Claude Code session to fix the error.

How it works

Midnight Ride hooks into Solid Errors and speaks up for three things: an error it hasn’t seen before, a resolved error that comes back, and, once a day, an unresolved error that’s still happening.

So I hear about each problem once, however often it happens.

Each alert also fires a Claude Code routine through its API trigger, at most five times a day. The session gets the error’s class, message, and backtrace, plus the commit that’s deployed. It writes a test that reproduces the error, fixes it, and opens a pull request to the app’s GitHub repo. If the error isn’t a bug in the app, it opens no pull request and says why. If a pull request for the error is already open, it stops.

The fix’s commit names the error plus its Solid Errors id (Resolves-Error: 90). When my server deploys that commit, it resolves the error in Solid Errors, so if the error comes back, I hear about it again. Errors that go quiet for a week resolve themselves.

A test run

To try it, I broke the filter for proper quotes in Quote Checker on purpose and deployed it. The error showed up on Telegram, a session opened a pull request with a test and a fix, I merged it, and the deploy resolved the error. All I did was read the pull request and click merge.

The part I had to think about

An error message can hold text an attacker chose. Someone could send a request that makes my app raise an error whose message reads like instructions to Claude. The session is told to treat the error as data, not instructions, but I don’t want to rely on that alone.

My first idea was the usual one: review every pull request, and protect the main branch so nothing gets merged without me. That doesn’t work here. Claude Code sessions reach GitHub as me, so GitHub can’t tell a session apart from me. A tricked session could merge its own pull request or push to main directly.

So my server no longer deploys whatever is on main. It deploys only commits marked by a tag signed with a local-only key, and it checks the signature before deploying. My deploy script signs the tag for me, so deploying is still one command. A session can push whatever it likes; without my key, none of it goes live.

I also keep the sessions’ network access limited to package registries and GitHub, so a tricked session can’t send my code anywhere. If you set up something like this, think about what a session can reach. Reviewing its work only helps if it can’t skip the review.

Try it

Midnight Ride is a gem for Rails apps that use Solid Errors:

gem "midnight_ride"

You’ll need a Telegram bot and, for the fixes, a Claude Code routine with an API trigger. The README walks through the setup, the routine’s prompt, and how to resolve errors on deploy. I run it on three apps: this blog, Quote Checker, and Veritula.