Skip to main content
HealOps uses Better Stack’s ClickHouse SQL Query API to read log evidence during investigations. It queries the configured source via remote(<source>_logs) for recent rows and s3Cluster(primary, <source>_s3) for historical rows, bounded by the alert window.

Prerequisites

  • A Better Stack account with at least one Telemetry source collecting logs
  • A ClickHouse HTTP client credential pair (username + password) generated from the dashboard
  • The region-specific query endpoint for your workspace (e.g. https://eu-nbg-2-connect.betterstackdata.com)
  • Network access from the HealOps environment to that endpoint over HTTPS

Setup

Option 1: Interactive CLI

You will be prompted for the query endpoint, username, password, and an optional comma-separated list of source IDs (planner hint).

Option 2: Environment variables

Add to your .env:

Option 3: Persistent store

Credentials are persisted to ~/.config/healops/integrations.json with 0o600 permissions:

Generating credentials

From the Better Stack dashboard:
  1. Open Telemetry and pick the source you want HealOps to query.
  2. In the source sidebar, open Integrations → Connect ClickHouse HTTP client.
  3. Copy the generated username, password, and query endpoint. The endpoint’s subdomain encodes the region (e.g. eu-nbg-2-connect, us-connect).
  4. The base source ID (e.g. t123456_myapp) is shown above the integration panel and is the value to supply for BETTERSTACK_SOURCES. Use the base name only — HealOps appends _logs and _s3 internally.

Investigation tool

HealOps exposes one tool against a Better Stack source:

query_betterstack_logs

Returns (dt, raw) pairs by UNIONing:
  • Recent rows from remote(<source>_logs)
  • Historical rows from s3Cluster(primary, <source>_s3) WHERE _row_type = 1
Arguments the planner supplies:
  • source — the base identifier (e.g. t123456_myapp). Falls back to the first configured sources entry when omitted.
  • since / until — ISO-8601 timestamps that bound the dt column. Optional, typically derived from the alert window.
  • limit — row cap; defaults to 500.
All queries run with FORMAT JSONEachRow and output_format_pretty_row_numbers=0. Source names are validated against ^[A-Za-z0-9_]+$ to prevent identifier injection.

Triggering investigation from an alert

When an alert carries a betterstack_source annotation, the planner wires it through to query_betterstack_logs automatically:

Verify

Expected output:
The verify step issues a cheap probe (SELECT 1 FORMAT JSONEachRow) against the configured endpoint using the stored credentials.

Troubleshooting

Security best practices

  • Use a dedicated ClickHouse HTTP client credential for HealOps — not your personal dashboard login.
  • Keep the credential pair out of source control — use .env or the persistent store (~/.config/healops/integrations.json).
  • The integration is read-only: HealOps only issues SELECT statements against remote(...) and s3Cluster(...) table functions.
  • Source identifiers are allowlisted against ^[A-Za-z0-9_]+$ before being interpolated into SQL, preventing identifier-injection attacks from the alert payload.
  • Rotate credentials periodically via the Better Stack dashboard.