Most of us build software with a safety net we don't think about: if something breaks badly enough, a human can step in. We can restart the service, fail over the database, ship a hotfix.

Spacecraft don't have that net. Nobody is driving out to fix a satellite, and even talking to one is limited by when it's in view of a ground station. So the space industry developed a discipline for handling failure on its own terms: FDIR, short for fault detection, isolation and recovery.

I find FDIR a useful lens for backend design, because it forces a question we usually skip: what should the system do by itself, before a human arrives?

Detect, isolate, recover

FDIR breaks failure handling into three steps:

1. Detect that something is wrong, usually by checking values against limits or by noticing that something expected didn't happen.
2. Isolate where the problem is, so you can act on the faulty part rather than the whole system.
3. Recover by switching to redundant hardware, restarting a component, or moving into a safer mode.

The sequence sounds obvious, but the key insight is that each step is designed in advance. Traditional FDIR relies on out-of-limit thresholds, and it often needs ground operators to isolate faults before recovery, which adds delay. That delay is exactly why spacecraft designers push as much of the loop as is safe onto the vehicle itself.

Faults have levels

One of the most useful ideas in FDIR is that not all faults deserve the same response. A published paper on onboard diagnosis describes a tiered scheme: the lowest levels are caught at the equipment level and fixed locally or by hardware redundancy, while more serious faults escalate to the onboard computer, and eventually to safe mode.

An industry presentation on FDIR across a family of satellite platforms frames the same idea from flight experience: recover in place and keep operating for the lower levels, and drop into safe mode, or a more minimal survival mode, only at the top.

That maps neatly onto a backend:

| Level | Spacecraft | Backend equivalent | |---|---|---| | Local | Unit fixes itself or switches to its redundant twin | Retry, circuit breaker, replica takes over | | Subsystem | Onboard computer reconfigures the affected subsystem | Feature disabled, queue paused, dependency bypassed | | System | Safe mode: minimal, stable, waits for the ground | Read-only mode, maintenance page, core path only |

The point of the table is that escalation is a design decision. If every fault goes straight to "page a human", you've only built one level.

Watchdogs and heartbeats

Spacecraft rely heavily on watchdogs: timers that something must keep resetting, and that trigger a recovery action when they're not reset. The same paper describes software faults that trip a watchdog being handled by rebooting the onboard software, with the satellite placed in safe mode if that doesn't resolve them. The Euclid space telescope's near-infrared instrument runs programmable watchdog checks to confirm its processing chains execute correctly and on time.

The backend version is a heartbeat with a consequence. Not just "emit a heartbeat metric", but "if the heartbeat stops, do something":

class Watchdog {
  private timer?: NodeJS.Timeout;

  constructor(
    private readonly timeoutMs: number,
    private readonly onExpire: () => Promise<void>,
  ) {}

  kick() {
    clearTimeout(this.timer);
    this.timer = setTimeout(() => {
      void this.onExpire();
    }, this.timeoutMs);
  }
}

// The worker must kick the watchdog after every processed batch.
const watchdog = new Watchdog(60_000, async () => {
  logger.error('worker stalled: restarting consumer');
  await consumer.restart();
});

The important part is the escalation path: if restarting doesn't help, what's the next level?

Safe mode is a designed state

When a spacecraft enters safe mode, it isn't crashing. It's moving into a deliberately designed state that keeps it alive and reachable while humans figure out what happened. When NASA's Chandra X-ray Observatory entered safe mode in February 2023, NASA said the likely cause was a sensor that monitors the position of the Sun providing incorrect data. The spacecraft paused science, moved to backup units, and turned to point perpendicular to the Sun while it waited for the ground. In 2015, a timing conflict in New Horizons' command sequence caused the spacecraft to switch to its backup computer and enter safe mode days before its Pluto flyby; the team recovered it, and none of the most critical observations were affected.

What's striking is how boring a good safe mode is. It does less, on purpose, so that it can keep doing it reliably.

Most backends don't have a safe mode; they have "up" and "down". Designing one usually means answering:

• What is the smallest useful thing this system must keep doing?
• Which features can we turn off to protect that?
• How do we enter this state automatically, and how do humans leave it?

In code, it often starts as nothing more than a well-named flag that every non-essential path respects:

type OperatingMode = 'nominal' | 'degraded' | 'safe';

export function canRun(feature: 'checkout' | 'recommendations' | 'exports', mode: OperatingMode) {
  if (mode === 'nominal') return true;
  if (mode === 'degraded') return feature !== 'exports';
  return feature === 'checkout'; // safe mode: only the core path
}

Leaving safe mode is a procedure too

The part of the Chandra story I find most instructive is the way out. Science didn't resume the moment the fault was understood. According to NASA's account, the team first addressed high temperatures that had reduced the sensitivity of the aspect camera the spacecraft uses to find stars and orient itself. They locked onto stars and set an accurate pointing direction, then spent several days manoeuvring between pointings to cool the camera and other critical components. Only then did they upload about a week of observations, with added restrictions meant to reduce the chance of another safe mode event. Science resumed roughly eleven days after the safe mode began.

That's a staged recovery: verify the basics, restore the foundations, resume a limited plan with extra guard rails, and widen it later.

Backends rarely recover that carefully. The usual pattern is to fix the bug and flip everything back on at once, which is how one incident turns into two. A staged exit looks more like this:

const recoverySteps = [
  { name: 'dependencies healthy', check: () => deps.allHealthy() },
  { name: 'core path at 5% traffic', check: () => canary.errorRate('checkout') < 0.01 },
  { name: 'core path at 100%', check: () => slo.ok('checkout') },
  { name: 'non-essential features on', check: () => slo.ok('recommendations') },
] as const;

async function leaveSafeMode() {
  for (const step of recoverySteps) {
    if (!(await step.check())) {
      throw new Error(`Recovery halted at: ${step.name}`); // stay where you are
    }
  }
}

The key property is that each step can stop the recovery without making things worse. If a check fails, you stay in a known state instead of sliding back into an incident.

Autonomy versus waiting for the ground

There's a tension in FDIR between letting the vehicle act on its own and waiting for operators, who have more context but may be hours away. Recent research explores onboard AI to detect, identify and suggest recovery actions, precisely because ground-in-the-loop recovery is slow.

Backend teams face the same trade-off with automated remediation. Automatic rollbacks, auto-scaling and circuit breakers are all forms of autonomy. The FDIR lesson is to be explicit about where the line sits: low-level, reversible actions can be automatic; actions that lose data or change the business should wait for a human.

A checklist I'd use

1. List your failure levels, from local to system-wide, and the automatic response at each.
2. Put a watchdog with a consequence on every long-running worker.
3. Design a safe mode, name it, and test entering and leaving it.
4. Keep automatic recovery to reversible actions; escalate the rest to humans.
5. Write down what "recovered" means, and leave safe mode in stages that can each stop safely.

You may never ship software to orbit, but designing as if nobody can reach your system in time is a good way to build one that rarely needs rescuing.