VOOZH about

URL: https://dev.to/lyraalishaikh/nightly-container-cve-gate-on-linux-trivy-systemd-timers-practical-auditable-29co

⇱ Nightly Container CVE Gate on Linux: Trivy + systemd Timers (Practical, Auditable) - DEV Community

Nightly Container CVE Gate on Linux: Trivy + systemd Timers (Practical, Auditable)

If you run containers on your own Linux host, the "it built last week" feeling can hide a lot of risk.

Images age fast. CVEs land daily. And if scanning only happens in CI (or never), you miss drift in long-lived deployments.

This guide builds a nightly CVE gate that:

  • scans selected container images on a schedule,
  • fails on high-impact findings,
  • writes JSON evidence you can audit later,
  • and uses systemd timers (not cron) for persistence and jitter.

No dashboards required — just reproducible files and logs.

What we are building

A small Linux-native pipeline:

  1. A Bash script scans each image with Trivy.
  2. Reports are saved to /var/log/container-cve-gate/.
  3. Exit status is non-zero if HIGH/CRITICAL vulns are found (policy gate).
  4. A systemd oneshot service runs the script.
  5. A systemd timer runs it nightly with Persistent=true and RandomizedDelaySec=.

Why this angle is different

I’ve already written about cron→systemd migration and self-hosted AI stacks. This post is narrower and more operational: continuous vulnerability drift detection for running image inventory, with report retention and policy gating.

Prerequisites

Example scan syntax and flags (--severity, --ignore-unfixed, --exit-code, JSON output) come from Trivy’s CLI docs.

Step 1) Define your image inventory

Create a plain text file with one image per line:

sudo install -d -m 0755 /etc/container-cve-gate
sudo tee /etc/container-cve-gate/images.txt >/dev/null <<'EOF'
nginx:1.27-alpine
redis:7-alpine
caddy:2
EOF

Keep this list small and intentional; these are the images you actually run.

Step 2) Create the scanner script 

sudo tee /usr/local/bin/container-cve-gate.sh >/dev/null <<'EOF'
#!/usr/bin/env bash
set -Eeuo pipefail

IMAGE_LIST="/etc/container-cve-gate/images.txt"
OUT_DIR="/var/log/container-cve-gate"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
RUN_DIR="$OUT_DIR/$STAMP"
SUMMARY="$RUN_DIR/summary.txt"

mkdir -p "$RUN_DIR"

if [[ ! -s "$IMAGE_LIST" ]]; then
 echo "ERROR: image list missing or empty: $IMAGE_LIST" >&2
 exit 2
fi

policy_failed=0

while IFS= read -r image; do
 [[ -z "$image" || "$image" =~ ^# ]] && continue

 safe_name="${image//[:\/]/_}"
 report_json="$RUN_DIR/${safe_name}.json"

 echo "Scanning: $image"

 # Trivy exits with code 1 when matching vulnerabilities are found
 # due to --exit-code 1.
 if trivy image \
 --severity HIGH,CRITICAL \
 --ignore-unfixed \
 --exit-code 1 \
 --format json \
 --output "$report_json" \
 "$image"; then
 status="PASS"
 else
 status="FAIL"
 policy_failed=1
 fi

 high_count=$(jq '[.. | objects | select(has("Severity")) | select(.Severity=="HIGH")] | length' "$report_json" 2>/dev/null || echo 0)
 critical_count=$(jq '[.. | objects | select(has("Severity")) | select(.Severity=="CRITICAL")] | length' "$report_json" 2>/dev/null || echo 0)

 printf "%s | HIGH=%s CRITICAL=%s | %s\n" "$image" "$high_count" "$critical_count" "$status" | tee -a "$SUMMARY"
done < "$IMAGE_LIST"

# Keep 30 days of report directories
find "$OUT_DIR" -mindepth 1 -maxdepth 1 -type d -mtime +30 -exec rm -rf {} +

if [[ "$policy_failed" -ne 0 ]]; then
 echo "Policy gate FAILED (HIGH/CRITICAL found). See: $RUN_DIR" >&2
 exit 1
fi

echo "Policy gate PASSED. See: $RUN_DIR"
EOF

sudo chmod 0755 /usr/local/bin/container-cve-gate.sh

Quick dry run 

sudo /usr/local/bin/container-cve-gate.sh
echo $?
  • 0 = no HIGH/CRITICAL findings under current policy
  • 1 = findings present (expected in many real environments)

Step 3) Add a systemd service 

sudo tee /etc/systemd/system/container-cve-gate.service >/dev/null <<'EOF'
[Unit]
Description=Nightly container CVE gate (Trivy)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/container-cve-gate.sh
# Hardening (minimal, practical)
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/container-cve-gate
ReadOnlyPaths=/etc/container-cve-gate
EOF

Step 4) Add a systemd timer (persistent + jitter) 

sudo tee /etc/systemd/system/container-cve-gate.timer >/dev/null <<'EOF'
[Unit]
Description=Run container CVE gate nightly

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=20m
AccuracySec=1m

[Install]
WantedBy=timers.target
EOF

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now container-cve-gate.timer

Validate schedule:

systemctl list-timers --all | grep container-cve-gate
systemctl cat container-cve-gate.timer

Why these fields matter:

  • Persistent=true: if the machine was down during schedule time, missed runs are triggered after boot.
  • RandomizedDelaySec=20m: spreads load and avoids synchronized thundering herd behavior.
  • AccuracySec=1m: keeps execution reasonably close to schedule.

Step 5) Audit and troubleshoot

Show recent service logs:

journalctl -u container-cve-gate.service -n 200 --no-pager

See last successful/failed runs:

systemctl status container-cve-gate.service
systemctl status container-cve-gate.timer

Inspect saved reports:

sudo find /var/log/container-cve-gate -maxdepth 2 -type f | sort
sudo cat /var/log/container-cve-gate/*/summary.txt | tail -n 20

Production tips

  1. Pin image tags (or digests) in inventory; avoid latest for policy signal quality.
  2. Separate policy from visibility: keep JSON reports even when the gate fails.
  3. Start strict where it counts: public-facing images first.
  4. Tune --ignore-unfixed intentionally: it reduces noise, but document that choice in team policy.
  5. Add alerting later: once baseline noise is known, send failed runs to your chat/on-call channel.

References

If you want, I can follow this with a part 2 that adds chat alerts + weekly trend reports while keeping the same Linux-native baseline.