Observability Setup - Completion Guide

✅ Current Status

Infrastructure Running

• ✅ ClickHouse (with authentication)
• ✅ Signoz OTEL Collector (fixed configuration)
• ✅ Signoz Query Service
• ✅ Signoz Frontend
• ✅ Infrastructure Nginx Proxy

Services Instrumented

• ✅ API Gateway (observability modules added)
• ✅ Booking Service (fully instrumented with custom spans and metrics)
• ⏳ Other services (ready to instrument - same pattern)

Next Steps

1. Verify Collector is Working

# Check collector logs
docker logs football-booking-backend-boilerplate-signoz-otel-collector-1

# Should see:
# {"level":"info","msg":"Everything is ready. Begin running and processing data."}

2. Test End-to-End Observability

Step 1: Start Your Services

# Start all services
yarn dev

# Or start individual services
yarn dev:gateway
yarn dev:booking

Step 2: Generate Some Traffic

# Make some API calls to generate traces
curl http://localhost:3000/api/health

# Or use the performance test script
yarn perf:test

Step 3: Check Signoz UI

1. Open Signoz: http://localhost:3301
2. Register/Login (should work now with the nginx proxy fix)
3. Check Traces:
4. Go to "Traces" tab
5. You should see traces from your services
6. Filter by service name (e.g., "api-gateway", "booking-service")
7. Check Metrics:
8. Go to "Metrics" tab
9. Query: http_requests_total
10. You should see request counts
11. Check Logs:
12. Go to "Logs" tab
13. Logs should be correlated with traces (trace ID visible)

3. Instrument Remaining Services

Follow the same pattern as Booking Service:

For Each Service:

1. Add to app.module.ts:

import {
  LoggerModule,
  MetricsInterceptor,
  MetricsModule,
  TraceInterceptor,
  TracingModule,
} from '@football-booking/observability';
import { APP_INTERCEPTOR } from '@nestjs/core';

@Module({
  imports: [
    // ... existing imports
    TracingModule,
    MetricsModule,
    LoggerModule,
  ],
  providers: [
    // ... existing providers
    {
      provide: APP_INTERCEPTOR,
      useClass: TraceInterceptor,
    },
    {
      provide: APP_INTERCEPTOR,
      useClass: MetricsInterceptor,
    },
  ],
})

1. Ensure main.ts initializes tracing (should already be done):

import { initializeTracing } from '@football-booking/observability';
initializeTracing('service-name'); // e.g., 'field-service', 'payment-service'

1. Add custom instrumentation (optional, for critical operations):

// In service files
constructor(
  // ... other dependencies
  private readonly traceService: TraceService,
  private readonly metricsService: MetricsService,
  private readonly loggerService: LoggerService,
) {}

async myOperation() {
  return await this.traceService.withSpan('my.operation', async (span) => {
    // Your business logic
    this.metricsService.recordSomething();
    return result;
  });
}

4. Import Dashboards

1. Go to Signoz UI: http://localhost:3301
2. Navigate to: Dashboards → Import
3. Import each dashboard:
4. config/signoz/dashboards/api-gateway-dashboard.json
5. config/signoz/dashboards/booking-service-dashboard.json
6. config/signoz/dashboards/database-dashboard.json
7. config/signoz/dashboards/kafka-dashboard.json

5. Configure Alerts (Optional)

1. Go to Signoz UI: http://localhost:3301
2. Navigate to: Alerts → Create Alert
3. Use the alert policies from config/signoz/alerts/alert-policies.yaml as reference
4. Configure notification channels (Slack, PagerDuty, etc.)

6. Verify Metrics Collection

Check that metrics are being collected:

# Query metrics via Signoz API
curl http://localhost:8083/api/v1/query?query=http_requests_total

# Or check in Signoz UI → Metrics

Verification Checklist

• [ ] Signoz UI accessible at http://localhost:3301
• [ ] Can register/login to Signoz
• [ ] OTEL Collector is running and healthy
• [ ] Traces visible in Signoz (after generating traffic)
• [ ] Metrics visible in Signoz
• [ ] Logs visible in Signoz (with trace correlation)
• [ ] Dashboards imported and showing data
• [ ] All services instrumented (or at least API Gateway + Booking Service)

Troubleshooting

No Traces Appearing

1. Check services are sending data:

bash # Check service logs for OpenTelemetry initialization # Should see: "✅ OpenTelemetry initialized for service: service-name"

1. Verify OTEL endpoint:

bash # Check environment variable echo $OTEL_EXPORTER_OTLP_ENDPOINT # Should be: http://localhost:4318

1. Check collector logs: bash docker logs football-booking-backend-boilerplate-signoz-otel-collector-1

Metrics Not Showing

1. Verify metrics are being recorded:
2. Check service code uses MetricsService

3. Check interceptors are registered

4. Check metric names:

5. Use exact names from code (e.g., http.requests.total)
6. Check Signoz UI → Metrics → Explore

Logs Not Correlated

1. Verify LoggerService is used:
2. Services should use LoggerService from observability lib

3. Not the default NestJS Logger

4. Check trace ID in logs:

5. Logs should include traceId and spanId fields
6. Should be JSON format

Quick Test Script

#!/bin/bash
# Quick test of observability stack

echo "1. Testing OTEL Collector..."
curl -s http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d '[]' && echo "✅ Collector responding"

echo "2. Testing Signoz Query Service..."
curl -s http://localhost:8083/api/health && echo "✅ Query service healthy"

echo "3. Testing Signoz Frontend..."
curl -s http://localhost:3301/health && echo "✅ Frontend accessible"

echo "4. Generating test traffic..."
curl -s http://localhost:3000/api/health && echo "✅ API Gateway responding"

echo "✅ All checks passed! Check Signoz UI at http://localhost:3301"

Summary

You now have:

• ✅ Complete observability infrastructure
• ✅ OpenTelemetry instrumentation
• ✅ Metrics collection (RED + business metrics)
• ✅ Structured logging with trace correlation
• ✅ Dashboards configured
• ✅ Alert policies defined
• ✅ SLO monitoring configured

Next: Instrument remaining services and start monitoring your application!