My role
Design, implementation and rollout (Tech Lead, Unix team)
Period
2026
Scale
SAP landscapes across Dev, QA and Prod on RHEL
Stack
Bash, sapcontrol, RHEL, systemd, Ansible
7ordered, health-gated steps per reboot
1 commandreplaces a manual, multi-team runbook
0steps that continue past a failed check

Context

Monthly OS patching on our RHEL estate included servers running SAP. The SAP subject-matter experts (SMEs) had a firm requirement: SAP layers must be stopped top-down — application servers, then central services, then the database — and started again bottom-up, with each layer confirmed healthy before the next one is touched.

In practice this meant a Unix engineer and an SAP engineer on a bridge call, stepping through a document, typing commands by hand on several hosts, and eyeballing output. It worked, but it was slow, depended on who was on shift, and the riskiest moment — starting application servers before the database was really ready — relied on human judgement at 2 a.m.

The problem, stated precisely

  • Order matters. Starting an application server before central services (message server and enqueue) are up leaves work processes in error; stopping the database first can leave in-flight updates in an inconsistent state.
  • “Command returned” is not “service is healthy”. sapcontrol ... Start returns as soon as the request is accepted. The real signal is the process list turning green.
  • Failure must stop the line. If any layer fails to stop or start cleanly, a human needs to look — automation that “pushes through” is worse than no automation.

Design

Ordered SAP stop and start around an OS reboot Stop phase: application servers, then central services, then database, each gated by a health check. Then the OS is patched and rebooted. Start phase runs in reverse: database, central services, application servers, then a final validation. STOP PHASE (top → down the stack) 1. App servers PAS / AAS instances 2. Central services ASCS / ERS 3. Database HANA / Oracle / etc. check check 4. Patch & reboot OS kernel, glibc… START PHASE (bottom → up the stack) 5. Database wait until open / online 6. Central services message + enqueue up 7. App servers work processes green check check Any failed check stops the run and pages the on-call engineer — the script never "pushes through".
The stop phase walks down the stack, the OS is patched, and the start phase walks back up. Every arrow is gated by a health check.

I wrote a Bash orchestrator with three principles:

  1. Declarative order, imperative execution. A small per-landscape config lists instances in stop order. Start order is simply the reverse — so there is only one list to keep correct.
  2. Every action is followed by a wait-and-verify. I used sapcontrol’s own exit codes rather than parsing text: GetProcessList returns 3 when all processes are running (GREEN) and 4 when all are stopped.
  3. Fail closed. Any unexpected state exits non-zero with a clear message, leaves the system where it is, and is picked up by the patching job as a failure for on-call to review.

A simplified, sanitised version of the core loop:

#!/usr/bin/env bash
# Ordered SAP stop — simplified extract. Real script adds logging, locking and a dry-run mode.
set -Eeuo pipefail
TIMEOUT=900   # seconds to wait for a layer to reach the target state
POLL=10
# Instances in STOP order: app servers -> central services -> database
# format: <host>:<instance_nr>:<label>
INSTANCES=(
  "app02:01:AAS"
  "app01:00:PAS"
  "cs01:10:ASCS"
  "db01:02:DB"
)
state_of() {  # echo 3 = all GREEN, 4 = all stopped, other = mixed/unknown
  local host=$1 nr=$2 rc=0
  ssh -o BatchMode=yes "$host" "sapcontrol -nr $nr -function GetProcessList" >/dev/null 2>&1 || rc=$?
  echo "$rc"
}
wait_for() {  # wait_for <host> <nr> <expected_rc>
  local host=$1 nr=$2 want=$3 waited=0
  until [[ "$(state_of "$host" "$nr")" == "$want" ]]; do
    (( waited >= TIMEOUT )) && { echo "FAIL: $host/$nr did not reach state $want in ${TIMEOUT}s" >&2; return 1; }
    sleep "$POLL"; (( waited += POLL ))
  done
}
for entry in "${INSTANCES[@]}"; do
  IFS=: read -r host nr label <<<"$entry"
  echo "Stopping $label on $host (nr $nr)"
  ssh -o BatchMode=yes "$host" "sapcontrol -nr $nr -function Stop" >/dev/null
  wait_for "$host" "$nr" 4      # gate: do not touch the next layer until this one is fully down
done
echo "All SAP layers stopped cleanly — safe to patch and reboot."

The start path walks the same array in reverse and waits for state 3 (all GREEN) at each step.

Alternatives I considered

Option Why not (for this estate)
sapcontrol -function StopSystem ALL from ASCS Handles ordering within one system, but didn’t cover the database layer or the SMEs’ requirement for explicit per-layer confirmation across hosts.
Pacemaker / HA cluster resource ordering Correct tool for HA-clustered SAP, but these systems were not clustered; introducing a cluster just for patching was out of proportion.
Ansible playbook only Ansible runs the script across hosts during patching, but the wait-and-verify logic is clearer and easier for SAP engineers to review as one readable script.

Rollout

  • Ran in dry-run mode (prints every command, executes none) alongside the manual process for a cycle so the SAP team could compare it line by line with their runbook.
  • Enabled on Dev, then QA, then Prod, one patch cycle each, with the SAP engineer still on the call.
  • Wired into the patching workflow so a non-zero exit blocks the reboot and raises an incident.

What I learned

  • The most valuable part was not the automation; it was writing the SMEs’ tacit knowledge down as an explicit, testable order and a definition of “healthy”.
  • Exit codes beat screen-scraping. If a vendor tool gives you a machine-readable status, build on that.
  • Automation earns trust through dry runs and side-by-side comparison, not through a demo.
  • Automation
  • SAP
  • Linux
  • Patching

My role
Tech Lead for the Unix team; owned monitor standards and the alert-to-ticket flow for Linux servers
Period
2024 – present
Scale
350+ servers · Dev / QA / Prod · on-prem and AWS EC2
Stack
Datadog, ServiceNow, Jira, RHEL, Rocky Linux, Ubuntu, AWS EC2
350+servers on one consistent alerting standard
1path from alert to ticket, with owner and runbook
Weeklyreview loop that retires noisy monitors

Context

Our estate is a mix of RHEL, Rocky Linux and Ubuntu servers spread across development, QA and production, both on-premises and on AWS EC2. Datadog monitors the servers and raises alerts; ServiceNow is the system of record for incidents; Jira tracks engineering work.

When monitoring grows organically, three problems appear: the same symptom fires several alerts, non-production noise looks as urgent as production, and tickets land without enough context for whoever is on call. The result is alert fatigue — and alert fatigue is how real incidents get missed.

Goals

  • Every ticket that reaches a human should need a human.
  • Every ticket should say what is wrong, where (environment and owner), and what to do first.
  • Recurring problems should be fixed permanently, not acknowledged forever.

Design

Alert-to-incident pipeline Datadog agents on 350+ servers send metrics and logs to Datadog monitors. Monitors with ownership tags and maintenance-window downtimes route actionable alerts to ServiceNow, which creates an incident assigned to the right group. The on-call engineer resolves it; recurring causes become Jira tasks, which feed back into tuning the monitors. 350+ servers RHEL · Rocky · Ubuntu on-prem + AWS EC2 Datadog Agent Datadog monitors owner + env tags composite conditions patch-window downtimes ServiceNow INC auto-created priority from env tag assignment group On-call engineer runbook link in INC restore service resolve + cause code informational / duplicate → dashboard only, no ticket Jira problem & permanent-fix tasks weekly review: tune or delete noisy monitors
One path from signal to ticket, with a feedback loop that keeps monitors honest.

1. Tag first, alert second. Monitors are only as routable as the metadata on the hosts. Every server carries consistent tags for environment, owning team and service. Priority in ServiceNow is derived from the environment tag, so a full disk in Dev does not page anyone at night.

2. Alert on symptoms that need action. Prefer one composite monitor (“filesystem > 90% and growing”) over several overlapping threshold alerts. Informational signals go to dashboards, not tickets.

3. Respect maintenance windows. Planned patching and reboots are covered by scheduled downtimes, so expected reboots don’t create incidents that someone then has to close as “planned work”.

4. Put the runbook in the ticket. Each monitor’s message includes a link to the relevant runbook section, so the first five minutes of response are the same whoever is on call.

5. Close the loop. Resolution cause codes feed a weekly review. Monitors that repeatedly create “no action required” tickets are tuned or deleted; recurring real causes become Jira tasks for a permanent fix.

Example: a better disk alert

A naive monitor fires the moment any filesystem crosses 90%. A better one combines level and trend, and carries the context the responder needs:

Title:   [{{env.name}}] {{host.name}} {{device.name}} filling up
Query (simplified):
         avg(last_30m):max:system.disk.in_use{env:prod} by {host,device} > 0.9
         AND forecast says full within 24h
Message: Filesystem is above 90% and trending up.
         Runbook: https://sol2lin.com/blog#disk-full-at-2am
         Owner:   {{team.name}}

The disk-full triage post is the runbook that alert links to.

What I learned

  • Alert quality is a process, not a configuration task. Without the weekly review, noise creeps back within months.
  • Tags are infrastructure. Inconsistent tagging silently breaks routing, priority and dashboards all at once.
  • The best measure of an alerting system is how the on-call engineer feels on Monday morning.
  • Observability
  • Incident Management
  • Linux
  • AWS