Incident Labs
Practice calm decisions under pressure
Incident response is not only technical troubleshooting. It is decision-making, communication, and prioritization under time pressure — often with incomplete information.
Incident Labs is scenario-based training for experienced developers and teams. You practice realistic incidents in a safe environment, and learn what good response looks like when systems misbehave.
What Incident Labs is
Incident Labs is a guided simulation of real-world incidents. Participants are given an evolving situation: symptoms, logs, stakeholder messages, and constraints. The team must decide what to do next, communicate clearly, and restore service — while balancing technical and business realities.
- realistic incident scenarios with branching paths
- time pressure and changing information
- technical decisions and trade-offs
- communication tasks (status updates, incident notes, stakeholder messages).
Example incident themes
- performance degradation and capacity limits
- partial outages and cascading failures
- data integrity problems and inconsistent reads
- deployment failures and rollback decisions
- timeouts, retries, and hidden amplification effects
- security-related incidents (scope-dependent).
The focus is not on hero debugging. The focus is on structured response, sound decisions, and clear communication.
How a session works
Sessions are run remotely (Teams) and are typically half-day or one-day workshops. The structure is intentionally practical:
- Briefing: system context, constraints, and roles
- Simulation: incident unfolds in phases, new information appears
- Decisions: participants choose actions and justify trade-offs
- Communication: written updates and short incident notes
- Review: what happened, why, and what would we change.
The goal is that participants leave with a clearer mental model of incident work: what to do first, what to avoid, and how to keep trust while solving the problem.
What you will practice
- triage and prioritization under uncertainty
- forming and testing hypotheses quickly
- rollback vs fix-forward decisions
- risk assessment and escalation
- writing clear status updates and incident summaries
- reducing “political” decision-making through clarity and evidence.
Who this is for
Incident Labs is for teams and individuals who already build and operate real systems. It works especially well for:
- senior developers and technical leads
- teams on call, or preparing for on-call responsibilities
- organizations with long-lived systems and real operational pressure.
Format and pricing
Incident Labs is typically delivered as:
- Half-day lab (remote, Teams)
- Full-day lab (remote, Teams)
- Custom series (multiple labs over time).
Pricing depends on scope, scenario design, and number of participants. If you describe your situation briefly, we can propose a sensible format.
If you want Incident Labs to reflect your own system, we can tailor scenarios based on your architecture and constraints.
Getting started
A good first step is a short conversation about your current incident practices, on-call setup, and the kinds of failures you want to prepare for.
Send us an email: info@intertechno.org
Further contact details can be found here.