Longer write-ups of work I owned end to end. Each one covers the business context, the constraints,
the design I chose (and the ones I rejected), how it was rolled out, and the measured result.
Replacing a fragile, multi-team manual runbook with a single script that stops and starts SAP layers in the order the SAP team requires — and refuses to continue if any layer is unhealthy.
Shaping how Datadog alerts become ServiceNow incidents for a hybrid RHEL, Rocky and Ubuntu estate, so on-call engineers get fewer, better tickets — each with an owner and a runbook.
Observability
Incident Management
Linux
AWS
Making SAP patch reboots boring: ordered, health-gated stop/start automation
Replacing a fragile, multi-team manual runbook with a single script that stops and starts SAP layers in the order the SAP team requires — and refuses to continue if any layer is unhealthy.
My role
Design, implementation and rollout (Tech Lead, Unix team)
Period
2026
Scale
SAP landscapes across Dev, QA and Prod on RHEL
Stack
Bash, sapcontrol, RHEL, systemd, Ansible
7ordered, health-gated steps per reboot
1 commandreplaces a manual, multi-team runbook
0steps that continue past a failed check
Context
Monthly OS patching on our RHEL estate included servers running SAP. The SAP subject-matter experts (SMEs) had a firm
requirement: SAP layers must be stopped top-down — application servers, then central services, then the database —
and started again bottom-up, with each layer confirmed healthy before the next one is touched.
In practice this meant a Unix engineer and an SAP engineer on a bridge call, stepping through a document, typing commands
by hand on several hosts, and eyeballing output. It worked, but it was slow, depended on who was on shift, and the riskiest
moment — starting application servers before the database was really ready — relied on human judgement at 2 a.m.
The problem, stated precisely
Order matters. Starting an application server before central services (message server and enqueue) are up leaves
work processes in error; stopping the database first can leave in-flight updates in an inconsistent state.
“Command returned” is not “service is healthy”.sapcontrol ... Start returns as soon as the request is accepted.
The real signal is the process list turning green.
Failure must stop the line. If any layer fails to stop or start cleanly, a human needs to look — automation that
“pushes through” is worse than no automation.
Design
The stop phase walks down the stack, the OS is patched, and the start phase walks back up. Every arrow is gated by a health check.
I wrote a Bash orchestrator with three principles:
Declarative order, imperative execution. A small per-landscape config lists instances in stop order. Start order
is simply the reverse — so there is only one list to keep correct.
Every action is followed by a wait-and-verify. I used sapcontrol’s own exit codes rather than parsing text:
GetProcessList returns 3 when all processes are running (GREEN) and 4 when all are stopped.
Fail closed. Any unexpected state exits non-zero with a clear message, leaves the system where it is, and is
picked up by the patching job as a failure for on-call to review.
A simplified, sanitised version of the core loop:
#!/usr/bin/env bash# Ordered SAP stop — simplified extract. Real script adds logging, locking and a dry-run mode.set-Eeuo pipefail
TIMEOUT=900# seconds to wait for a layer to reach the target statePOLL=10# Instances in STOP order: app servers -> central services -> database# format: <host>:<instance_nr>:<label>INSTANCES=("app02:01:AAS""app01:00:PAS""cs01:10:ASCS""db01:02:DB")state_of(){# echo 3 = all GREEN, 4 = all stopped, other = mixed/unknownlocalhost=$1nr=$2rc=0ssh-oBatchMode=yes "$host""sapcontrol -nr $nr -function GetProcessList">/dev/null 2>&1||rc=$?echo"$rc"}wait_for(){# wait_for <host> <nr> <expected_rc>localhost=$1nr=$2want=$3waited=0until[["$(state_of "$host""$nr")"=="$want"]];do(( waited >= TIMEOUT ))&&{echo"FAIL: $host/$nr did not reach state $want in ${TIMEOUT}s">&2;return1;}sleep"$POLL";(( waited += POLL ))done}forentryin"${INSTANCES[@]}";doIFS=: read-rhost nr label <<<"$entry"echo"Stopping $label on $host (nr $nr)"ssh-oBatchMode=yes "$host""sapcontrol -nr $nr -function Stop">/dev/null
wait_for "$host""$nr"4# gate: do not touch the next layer until this one is fully downdoneecho"All SAP layers stopped cleanly — safe to patch and reboot."
The start path walks the same array in reverse and waits for state 3 (all GREEN) at each step.
Alternatives I considered
Option
Why not (for this estate)
sapcontrol -function StopSystem ALL from ASCS
Handles ordering within one system, but didn’t cover the database layer or the SMEs’ requirement for explicit per-layer confirmation across hosts.
Pacemaker / HA cluster resource ordering
Correct tool for HA-clustered SAP, but these systems were not clustered; introducing a cluster just for patching was out of proportion.
Ansible playbook only
Ansible runs the script across hosts during patching, but the wait-and-verify logic is clearer and easier for SAP engineers to review as one readable script.
Rollout
Ran in dry-run mode (prints every command, executes none) alongside the manual process for a cycle so the SAP team
could compare it line by line with their runbook.
Enabled on Dev, then QA, then Prod, one patch cycle each, with the SAP engineer still on the call.
Wired into the patching workflow so a non-zero exit blocks the reboot and raises an incident.
What I learned
The most valuable part was not the automation; it was writing the SMEs’ tacit knowledge down as an explicit, testable
order and a definition of “healthy”.
Exit codes beat screen-scraping. If a vendor tool gives you a machine-readable status, build on that.
Automation earns trust through dry runs and side-by-side comparison, not through a demo.
From alert noise to actionable incidents across 350+ servers
Shaping how Datadog alerts become ServiceNow incidents for a hybrid RHEL, Rocky and Ubuntu estate, so on-call engineers get fewer, better tickets — each with an owner and a runbook.
My role
Tech Lead for the Unix team; owned monitor standards and the alert-to-ticket flow for Linux servers
Period
2024 – present
Scale
350+ servers · Dev / QA / Prod · on-prem and AWS EC2
1path from alert to ticket, with owner and runbook
Weeklyreview loop that retires noisy monitors
Context
Our estate is a mix of RHEL, Rocky Linux and Ubuntu servers spread across development, QA and production, both
on-premises and on AWS EC2. Datadog monitors the servers and raises alerts; ServiceNow is the system of record for
incidents; Jira tracks engineering work.
When monitoring grows organically, three problems appear: the same symptom fires several alerts, non-production noise
looks as urgent as production, and tickets land without enough context for whoever is on call. The result is alert
fatigue — and alert fatigue is how real incidents get missed.
Goals
Every ticket that reaches a human should need a human.
Every ticket should say what is wrong, where (environment and owner), and what to do first.
Recurring problems should be fixed permanently, not acknowledged forever.
Design
One path from signal to ticket, with a feedback loop that keeps monitors honest.
1. Tag first, alert second. Monitors are only as routable as the metadata on the hosts. Every server carries
consistent tags for environment, owning team and service. Priority in ServiceNow is derived from the environment tag,
so a full disk in Dev does not page anyone at night.
2. Alert on symptoms that need action. Prefer one composite monitor (“filesystem > 90% and growing”) over several
overlapping threshold alerts. Informational signals go to dashboards, not tickets.
3. Respect maintenance windows. Planned patching and reboots are covered by scheduled downtimes, so expected
reboots don’t create incidents that someone then has to close as “planned work”.
4. Put the runbook in the ticket. Each monitor’s message includes a link to the relevant runbook section, so the
first five minutes of response are the same whoever is on call.
5. Close the loop. Resolution cause codes feed a weekly review. Monitors that repeatedly create “no action
required” tickets are tuned or deleted; recurring real causes become Jira tasks for a permanent fix.
Example: a better disk alert
A naive monitor fires the moment any filesystem crosses 90%. A better one combines level and trend, and carries the
context the responder needs:
Title: [{{env.name}}] {{host.name}} {{device.name}} filling up
Query (simplified):
avg(last_30m):max:system.disk.in_use{env:prod} by {host,device} > 0.9
AND forecast says full within 24h
Message: Filesystem is above 90% and trending up.
Runbook: https://sol2lin.com/blog#disk-full-at-2am
Owner: {{team.name}}