TL;DR

The existing AWS Blog approach ships FSx for ONTAP audit logs to Splunk via two EC2 instances (syslog-ng + Universal Forwarder). We replaced it with a single Lambda function — same Splunk index, same SPL queries, 90% AWS infrastructure cost reduction.

[Before] FSx for ONTAP → syslog-ng (EC2) → Splunk UF (EC2) → Splunk
 Monthly AWS infra cost: ~$66 (2× t3.medium + EBS)
 Ops burden: OS patching, agent updates, scaling

[After] FSx for ONTAP → S3 Access Point → Lambda → Splunk HEC
 Monthly AWS infra cost: ~$6 (Lambda + EventBridge)
 Ops burden: Zero (managed services only)

Important: The 90% cost reduction refers to AWS infrastructure costs only (EC2/Lambda/EventBridge). Splunk platform licensing costs remain unchanged regardless of the delivery method.

This is Part 8 of the Serverless Observability for FSx for ONTAP series.

The Problem with EC2-Based Splunk Integration

The AWS Blog's architecture works, but it comes with operational overhead:

If you're already running this EC2 pattern and want to modernize, this article shows you how — with a parallel deployment strategy that ensures zero data loss during cutover.

Architecture

┌──────────────────────────────────────────────────────────┐
│ FSx for ONTAP │
│ │
│ Audit Volume ──→ S3 Access Point │
│ │ │
│ ▼ │
│ EventBridge Scheduler (rate: 5 min) │
│ │ │
│ ▼ │
│ Lambda (Python 3.12) │
│ • Reads audit logs via S3 AP │
│ • Parses JSON/EVTX │
│ • Formats as Splunk HEC events │
│ • Sends with Authorization: Splunk <token> │
│ • Checkpoints in SSM Parameter Store │
│ │ │
│ ▼ │
│ Splunk HEC │
│ https://<splunk>:8088/services/collector/event │
│ Response: {"text":"Success","code":0} │
│ │
│ SPL: index=fsxn_audit sourcetype=fsxn:ontap:audit │
└──────────────────────────────────────────────────────────┘

High-Volume Alternative: Firehose Path

For sustained >1000 events/sec, use Kinesis Data Firehose with its built-in Splunk destination:

FSx for ONTAP → S3 AP → Lambda (transform) → Kinesis Data Firehose → Splunk HEC

A separate template-firehose.yaml is provided for this path.

Migration Strategy (Zero Data Loss)

Phase 1: Parallel Deployment (Day 1-3)

Deploy the serverless stack alongside the existing EC2 pipeline. Use a separate Splunk index for validation:

aws cloudformation deploy \
 --template-file integrations/splunk-serverless/template.yaml \
 --stack-name fsxn-splunk-integration \
 --parameter-overrides \
 S3AccessPointArn=<S3_AP_ARN> \
 SplunkHecTokenSecretArn=<SECRET_ARN> \
 SplunkHecEndpoint=https://splunk.example.com:8088 \
 S3BucketName=<BUCKET> \
 SplunkIndex=fsxn_audit_serverless \
 --capabilities CAPABILITY_IAM

Compare events between old and new pipelines for 48 hours:

| stats count by index
| where index IN ("fsxn_audit", "fsxn_audit_serverless")

Phase 2: Cutover (Day 4-5)

Once event parity is confirmed:

1. Update the stack to use the production index (fsxn_audit)
2. Stop the syslog-ng and UF services on EC2 (don't terminate yet)
3. Monitor for 24 hours

Phase 3: Cleanup (Day 7+)

# Terminate EC2 instances
# Remove security groups, IAM roles, EBS volumes
# Delete old CloudFormation/Terraform resources

What Changes for Splunk Users

Unchanged ✅

• Index name and sourcetype (configurable)
• SPL queries — same field names
• Dashboards and saved searches
• Alert rules

Changed ⚠️

• host field: EC2 hostname → SVM name
• source field: syslog path → fsxn-observability
• Delivery latency: near-real-time (syslog) → polling interval (default 5 min)

HEC Event Format

{"time":1716508800,"host":"svm-prod-01","source":"fsxn-observability","sourcetype":"fsxn:ontap:audit","index":"fsxn_audit","event":{"event_type":"4663","user":"admin@corp.local","operation":"ReadData","path":"/vol/data/report.pdf","result":"Success","client_ip":"10.0.1.50"}}

SPL Query Examples

# Failed access attempts
index=fsxn_audit sourcetype=fsxn:ontap:audit result=Failure
| stats count by user, path
| sort -count

# Operations timeline
index=fsxn_audit sourcetype=fsxn:ontap:audit
| timechart span=5m count by operation

# Top users
index=fsxn_audit sourcetype=fsxn:ontap:audit
| stats count by user
| sort -count
| head 20

# Specific user investigation
index=fsxn_audit sourcetype=fsxn:ontap:audit user="admin@corp.local"
| table _time, operation, path, result, client_ip

Cost Comparison

Note: EC2 cost assumes 2× t3.medium (as per the AWS Blog reference architecture). Actual EC2 costs vary by instance type and region. Splunk Cloud licensing costs are contract-dependent and may differ significantly from list pricing.

Network Considerations

⚠️ VerifySSL: Set to true in production. Only use false for self-signed certs in dev environments.

Rollback Plan

If issues are discovered after cutover:

1. Start the stopped EC2 instances (syslog-ng + UF)
2. Verify syslog-ng is receiving events
3. Delete the serverless CloudFormation stack
4. Investigate and resolve before re-attempting

The serverless Lambda uses checkpointing — no events are lost during the overlap period (brief duplicates are possible).

What's Next

• Firehose path: For high-volume logs (>1000 events/sec), use template-firehose.yaml
• HEC Acknowledgment (useACK): For Level 2+, enable HEC indexer acknowledgment to guarantee at-least-once delivery. Lambda waits for ack before advancing checkpoint
• CIM compliance: Map fields to Splunk's Common Information Model (Authentication or Change data model) for compatibility with Splunk Enterprise Security correlation searches
• Index pre-creation: The fsxn_audit index must be created before first ingestion (Splunk Cloud: Admin Console; Enterprise: indexes.conf)
• EMS webhooks: Real-time ARP ransomware detection alerts
• FPolicy: Sub-second file operation streaming
• Production Readiness: Progress from Level 1 (this Quick Start) to Level 4 (Enterprise) — see the Pipeline SLO Definitions

Production Readiness

This integration follows the project's Production Readiness Levels:

Data classification: Splunk receives user and path fields (PII/sensitive). For Splunk Cloud, data is processed in the vendor's infrastructure. For self-hosted Splunk Enterprise, data stays in your VPC. See Data Classification Guide for field-by-field PII classification and handling patterns.

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

Resources

• GitHub: fsxn-observability-integrations/integrations/splunk-serverless
• Migration Guide (detailed)
• AWS Blog: EC2-based approach
• Splunk HEC Documentation
• Pipeline SLO Definitions
• Data Classification Guide

Series Navigation

• Part 1: Why Your FSx for ONTAP Audit Logs Deserve Better Than EC2
• Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
• Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
• Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
• Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
• Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
• Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
• Part 8: EC2 to Serverless: Modernizing Splunk Integration (this post)
• Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
• Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
• Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
• Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
• Part 13: 9 Vendors, One Architecture

Questions about the Splunk migration or serverless HEC delivery? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations