Bỏ qua

Signoz OTEL Collector Configuration Fix

Problem

The Signoz OTEL collector was failing to start with this error:

failed to start collector service: failed to start : failed to get config:
cannot resolve the configuration: invalid uri: "DEPLOYMENT_ENVIRONMENT:development"

Root Cause

The Signoz OTEL collector does not support environment variable substitution in YAML configuration files using the ${VAR:default} syntax. When it encountered:

resource:
  attributes:
    - key: deployment.environment
      value: ${DEPLOYMENT_ENVIRONMENT:development}

It tried to use the literal string "DEPLOYMENT_ENVIRONMENT:development" as a value, which caused the configuration error.

Solution

Option 1: Use Hardcoded Value (Current Fix)

Changed the resource processor to use a hardcoded value:

resource:
  attributes:
    - key: deployment.environment
      value: development # Hardcoded for now
      action: upsert

Note: The deployment environment is actually set by each service's OpenTelemetry SDK initialization (in libs/observability/src/tracing/tracing.ts), so this collector-level setting is redundant anyway.

Option 2: Remove Resource Processor (Alternative)

If you don't need the resource processor, you can remove it entirely:

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
    check_interval: 1s
  # Resource processor removed - environment set by services

And update the pipelines:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch] # Removed 'resource'
      exporters: [clickhousetraces]

Why This Works

  1. Services set environment: Each service (api-gateway, booking-service, etc.) sets the deployment environment when initializing OpenTelemetry:

typescript resource: new Resource({ [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env['NODE_ENV'] || 'development', });

  1. Collector doesn't need it: The collector just forwards traces/metrics/logs. The environment attribute is already set by the services.

  2. Hardcoded is fine: For development, hardcoding development is acceptable. For production, you can:

  3. Use different config files per environment
  4. Or rely on services to set the correct environment

Changes Made

  1. Fixed otel-collector-config.yaml:
  2. Changed ${DEPLOYMENT_ENVIRONMENT:development} → development

  3. Added comment explaining why

  4. Updated docker-compose.dev.yml:

  5. Added DEPLOYMENT_ENVIRONMENT environment variable to collector container
  6. This is for reference, not used by the config

Verification

After the fix, restart the collector:

docker compose -f docker-compose.dev.yml restart signoz-otel-collector

Check logs:

docker logs signoz-otel-collector

Should see:

{"level":"info","msg":"Starting collector service"}
{"level":"info","msg":"Everything is ready. Begin running and processing data."}

Production Considerations

For production, you have a few options:

Option 1: Separate Config Files

config/
├── signoz/
│   ├── otel-collector-config.dev.yaml
│   └── otel-collector-config.prod.yaml

Option 2: Use Docker Compose Overrides

# docker-compose.prod.yml
services:
  signoz-otel-collector:
    volumes:
      - ./config/signoz/otel-collector-config.prod.yaml:/etc/otel-collector-config.yaml

Option 3: Rely on Service-Level Configuration

Since services already set the environment attribute, you can just remove the resource processor from the collector config entirely.

References