Splunk

HealOps queries Splunk using the REST API to surface relevant log evidence during alert investigations — searching indexes with SPL, correlating error patterns with incidents, and identifying root causes.

Prerequisites

Splunk Enterprise or Splunk Cloud instance (version 8.x or later)
REST API access on port 8089
A bearer token with search capability (see Generating a Bearer Token)

Setup

Option 1: Interactive CLI

healops integrations setup

Select Splunk when prompted and provide your REST API base URL and bearer token.

Option 2: Environment variables

Add to your .env:

SPLUNK_URL=https://splunk.corp.com:8089    # REST API base URL (port 8089 default)
SPLUNK_TOKEN=your-bearer-token             # API bearer token (NOT an HEC token)
SPLUNK_INDEX=main                          # Default index to search (optional)
SPLUNK_VERIFY_SSL=true                     # Set false to skip SSL verification (optional)
SPLUNK_CA_BUNDLE=/etc/ssl/certs/corp-ca.pem  # Path to custom CA bundle (optional)

Variable	Default	Description
`SPLUNK_URL`	—	Required. REST API base URL including port
`SPLUNK_TOKEN`	—	Required. Bearer token with search capability
`SPLUNK_INDEX`	`main`	Default index searched when no index is specified in the alert
`SPLUNK_VERIFY_SSL`	`true`	Set to `false` to disable SSL verification (dev/local only)
`SPLUNK_CA_BUNDLE`	—	Path to a PEM CA bundle for enterprise self-signed certificates

Option 3: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "splunk-prod",
      "service": "splunk",
      "status": "active",
      "credentials": {
        "base_url": "https://splunk.corp.com:8089",
        "token": "your-bearer-token",
        "index": "main",
        "verify_ssl": true,
        "ca_bundle": "/etc/ssl/certs/corp-ca.pem"
      }
    }
  ]
}

Multi-instance setup

To connect multiple Splunk instances (e.g. separate prod and staging clusters):

SPLUNK_INSTANCES='[
  {"name":"prod","tags":{"env":"prod"},"credentials":{"base_url":"https://splunk-prod:8089","token":"prod-token","index":"prod"}},
  {"name":"staging","tags":{"env":"staging"},"credentials":{"base_url":"https://splunk-staging:8089","token":"staging-token","index":"staging"}}
]'

When SPLUNK_INSTANCES is set it overrides the single-instance SPLUNK_URL / SPLUNK_TOKEN variables.

Generating a bearer token

HealOps uses bearer tokens — not basic auth and not HEC tokens. To generate one: Via the Splunk UI:

Go to Settings → Tokens
Click New Token
Set a name (e.g. healops) and an expiry date
Copy the generated token

Via the REST API (replace <PASSWORD> with your admin password):

curl -sk -u admin:<PASSWORD> \
  https://splunk.corp.com:8089/services/authorization/tokens \
  -X POST \
  --data-urlencode "name=healops" \
  --data-urlencode "expires_on=+90d" \
  --data-urlencode "output_mode=json" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['entry'][0]['content']['token'])"

The token needs the search capability. The admin role includes this by default. For a dedicated service account, ensure the role includes:

search
read_splunkd_private_settings (needed for the verify call against /services/server/info)

Verify

healops integrations verify splunk

Expected output:

Service: splunk
Status: passed
Detail: Connected to Splunk 9.x.x

How queries are generated

HealOps builds SPL queries deterministically from the alert payload — the LLM selects which tool to call but never writes the query itself. This keeps investigations reproducible and auditable. Query construction priority:

Priority	Source	Example
1	`annotations.splunk_query` — verbatim SPL from your alert	`index=prod "PaymentTimeout" \| head 50`
2	`annotations.query` or `annotations.log_query`	Any pre-populated query field
3	`error_message` field — keyword search built automatically	`search index=main "NullPointerException" \| head 50`
4	`alert_name` — last-resort keyword search	`search index=main "payments-error-spike" \| head 50`
5	Fallback — index scan	`search index=main \| head 50`

To pass a specific SPL query through an alert, set commonAnnotations.splunk_query:

{
  "alert_name": "Payment service errors",
  "commonAnnotations": {
    "splunk_query": "index=prod sourcetype=app_logs \"NullPointerException\" | head 50"
  }
}

Test with the built-in alert template

healops investigate --template splunk

This runs a synthetic investigation using a pre-built alert fixture — no live alert infrastructure needed.

Troubleshooting

Symptom	Fix
`SSL: CERTIFICATE_VERIFY_FAILED`	Set `SPLUNK_CA_BUNDLE=/path/to/corp-ca.pem` (preferred) or `SPLUNK_VERIFY_SSL=false` (dev only)
`HTTP 401 Unauthorized`	Token expired or was generated with the wrong account — regenerate
`HTTP 403 Forbidden`	Token lacks `search` capability — check the role assigned to the token
Empty search results	Data may not have been ingested yet, or the index name is wrong
`Connection refused` on port 8089	Splunk management port may be firewalled; confirm network access
`healops integrations verify` fails	Check `SPLUNK_URL` includes the protocol and port (`https://host:8089`)

Security best practices

Use a read-only bearer token — never use an admin token in production.
Store SPLUNK_TOKEN in .env or the credential store, not in source code or CI logs.
Prefer a dedicated healops service account with only the search capability.
For enterprise self-signed certificates, set SPLUNK_CA_BUNDLE to the CA bundle path rather than disabling verification entirely.
Set SPLUNK_VERIFY_SSL=false only in local or dev environments when you cannot supply a CA bundle.
Rotate tokens on a schedule and revoke them when no longer needed.

Overview

LLM providers

Observability and incidents

Cloud, code, and collaboration

Messaging

Data and workflow systems

Prerequisites

Setup

Option 1: Interactive CLI

Option 2: Environment variables

Option 3: Persistent store

Multi-instance setup

Generating a bearer token

Verify

How queries are generated

Test with the built-in alert template

Troubleshooting

Security best practices

​Prerequisites

​Setup

​Option 1: Interactive CLI

​Option 2: Environment variables

​Option 3: Persistent store

​Multi-instance setup

​Generating a bearer token

​Verify

​How queries are generated

​Test with the built-in alert template

​Troubleshooting

​Security best practices

Prerequisites

Setup

Option 1: Interactive CLI

Option 2: Environment variables

Option 3: Persistent store

Multi-instance setup

Generating a bearer token

Verify

How queries are generated

Test with the built-in alert template

Troubleshooting

Security best practices