AWS Batch

Try Tracer in Your AWS Batch Workloads

CloudFormation
Terraform

Use CloudFormation to set up AWS Batch infrastructure for bioinformatics and HPC pipelines.

1. Connect AWS Integration

Open the following link to connect your AWS account automatically to Tracer. This will allow Tracer to monitor your AWS Batch jobs.→ Connect AWS

2. Locate the EC2 LaunchTemplate UserData

In your CloudFormation template, find the UserData field within the Launch Template used by your AWS Batch compute environment.Reference example: CloudFormation Template

Resources:
  NextflowLaunchTemplate:
    Properties:
      LaunchTemplateData:
        UserData: !Base64 |
          #cloud-config
          runcmd:
            - ...

3. Append Tracer install in EC2 Launch Template UserData

Add these commands to the runcmd: section in your UserData:

- curl -sSL https://install.healops.ai | sh

- tracer init --token YOUR_TOKEN_HERE \
     --pipeline-name YOUR_PIPELINE_NAME \
     --pipeline-type YOUR_PIPELINE_TYPE \
     --environment aws_batch \
     --watch-dir="/var/log"

Go to our onboarding to get your own personal token

4. Validate, apply change set, then verify instance

aws cloudformation validate-template \
  --template-body file://YOUR_TEMPLATE.yaml

aws cloudformation create-change-set \
  --stack-name YOUR_STACK_NAME \
  --template-body file://YOUR_TEMPLATE.yaml \
  --change-set-name tracer-userdata-update \
  --capabilities CAPABILITY_NAMED_IAM

aws cloudformation execute-change-set \
  --stack-name YOUR_STACK_NAME \
  --change-set-name tracer-userdata-update

5. Configure Nextflow batch.config with Trace ID

In your Nextflow batch.config file, you need to:

Add a params section to generate a UUID that will be shared across all jobs in the workflow
Add containerOptions in the process section to pass the trace ID as an environment variable to each container

This allows Tracer to correlate all jobs from the same workflow execution.

// 1. Add params section to generate UUID
params {customUUID = java.util.UUID.randomUUID().toString()}

process {executor = 'awsbatch'
    queue = 'NextflowCPU'

// 2. Add containerOptions to pass trace ID to containers
containerOptions = "--env TRACER_TRACE_ID=${params.customUUID}"

resourceLabels = ['custom-session-uuid': "${params.customUUID}"]
}

aws {region = 'YOUR_REGION'
  batch
  {cliPath = '/usr/local/aws-cli/v2/current/bin/aws'}
}

Monitor and Optimize Your Pipeline

Watch your pipeline run in the Tracer dashboard

View real-time metrics, resource usage, and performance insights for your pipeline runs.

Use Terraform to provision infrastructure and run RNA-seq pipeline on AWS Batch.

Prerequisites

Make sure these tools are installed locally before proceeding.

• AWS CLI v2

• Terraform

• Nextflow

• Java

• Docker (optional)

1. Connect AWS Integration

Open the following link to connect your AWS account automatically to Tracer. This will allow Tracer to monitor your AWS Batch jobs.→ Connect AWS

2. Clone the repository

Clone the nextflow-test-pipelines repository to get access to the AWS Batch RNA-seq pipeline configuration.

git clone https://github.com/healops-ai/nextflow-test-pipelines.git
cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq

3. Update user_data_mime.sh

Update the user_data_mime.sh file to include your Tracer user ID. Navigate to the terraform directory and edit the file:

cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq/terraform
vim user_data_mime.sh

Important: Find line 120 in the file and update it with your user ID:

tracer init --token YOUR_TOKEN_HERE
--pipeline-name aws_batch_rnaseq
--pipeline-type RNA-SEQ
--environment aws-batch
--watch-dir="/var/log"

Go to our onboarding to get your own personal token

Note: If you need to change this later, you’ll need to refresh the infrastructure.

View the exact line to update

4. Setup infrastructure

Run the setup script to create the AWS infrastructure (VPC, S3, Batch, etc.):

cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq
./run.sh setup

What this does: Creates VPC with public/private subnets, AWS Batch compute environment, S3 buckets for work directory and outputs, IAM roles and policies, security groups, and EC2 Instance Connect endpoint.

Alternative for Tracer AWS Account Users: If you have access to Tracer’s AWS account, you can skip the infrastructure setup and run the pipeline directly: bash cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq ./run.sh tracer run

5. Run the RNA-seq pipeline

Execute the RNA-seq pipeline using the run script:

cd nextflow-test-pipelines/pipelines/aws-batch/rnaseq
./run.sh run

Monitor and Optimize Your Pipeline

Watch your pipeline run in the Tracer dashboard

View real-time metrics, resource usage, and performance insights for your pipeline runs.

Getting started

Use cases

Tutorials

Frameworks

Technology

Deployment environments

Comparisons

1. Connect AWS Integration

2. Locate the EC2 LaunchTemplate UserData

3. Append Tracer install in EC2 Launch Template UserData

4. Validate, apply change set, then verify instance

5. Configure Nextflow batch.config with Trace ID

Monitor and Optimize Your Pipeline

1. Connect AWS Integration

2. Clone the repository

3. Update user_data_mime.sh

4. Setup infrastructure

5. Run the RNA-seq pipeline

Monitor and Optimize Your Pipeline

​1. Connect AWS Integration

​2. Locate the EC2 LaunchTemplate UserData

​3. Append Tracer install in EC2 Launch Template UserData

​4. Validate, apply change set, then verify instance

​5. Configure Nextflow batch.config with Trace ID

​Monitor and Optimize Your Pipeline

​1. Connect AWS Integration

​2. Clone the repository

​3. Update user_data_mime.sh

​4. Setup infrastructure

​5. Run the RNA-seq pipeline

​Monitor and Optimize Your Pipeline

1. Connect AWS Integration

2. Locate the EC2 LaunchTemplate UserData

3. Append Tracer install in EC2 Launch Template UserData

4. Validate, apply change set, then verify instance

5. Configure Nextflow batch.config with Trace ID

Monitor and Optimize Your Pipeline

1. Connect AWS Integration

2. Clone the repository

3. Update user_data_mime.sh

4. Setup infrastructure

5. Run the RNA-seq pipeline

Monitor and Optimize Your Pipeline