Observability Implementation Guide

Overview

This document describes the observability implementation for the Football Booking Backend system using OpenTelemetry and Signoz.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Services (NestJS)                        │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐     │
│  │ API      │  │ Booking │  │ Payment │  │ Field   │     │
│  │ Gateway  │  │ Service │  │ Service │  │ Service │     │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └────┬─────┘     │
│       │            │             │             │            │
│       └────────────┴─────────────┴─────────────┘            │
│                    │                                          │
│              OpenTelemetry SDK                                │
│              (Traces, Metrics, Logs)                         │
└────────────────────┬──────────────────────────────────────────┘
                     │
                     │ OTLP (gRPC/HTTP)
                     │
┌────────────────────▼──────────────────────────────────────────┐
│              Signoz OTEL Collector                            │
│              (Port 4317/4318)                                 │
└────────────────────┬──────────────────────────────────────────┘
                     │
                     │
┌────────────────────▼──────────────────────────────────────────┐
│              ClickHouse Database                              │
│              (Stores traces, metrics, logs)                  │
└────────────────────┬──────────────────────────────────────────┘
                     │
┌────────────────────▼──────────────────────────────────────────┐
│              Signoz Query Service + Frontend                  │
│              (Port 8083/3301)                                 │
│              - Dashboards                                     │
│              - Alerts                                         │
│              - Trace Explorer                                 │
└───────────────────────────────────────────────────────────────┘

Components

1. OpenTelemetry SDK

Initialized in each service's main.ts before any other imports:

import { initializeTracing } from '@football-booking/observability';
initializeTracing('service-name');

2. Observability Library (`libs/observability`)

Tracing Module

TraceService: Create custom spans for business operations
TraceInterceptor: Automatic span creation for HTTP requests
Trace Decorator: @Trace() decorator for methods

Metrics Module

MetricsService: Collect RED metrics (Rate, Errors, Duration)
MetricsInterceptor: Automatic HTTP metrics collection
Business metrics: bookings, payments, notifications
Infrastructure metrics: DB, Redis, Kafka

Logging Module

LoggerService: Structured JSON logging with trace correlation
Automatically includes trace ID and span ID in logs

Metrics Collected

RED Metrics (Rate, Errors, Duration)

HTTP Requests

http.requests.total - Total HTTP requests
http.requests.errors - Error requests (4xx, 5xx)
http.request.duration - Request latency histogram

Database Queries

db.queries.total - Total database queries
db.queries.errors - Query errors
db.query.duration - Query latency histogram
db.connections.active - Active connections
db.connections.usage_percent - Connection pool usage

Redis Operations

redis.operations.total - Total Redis operations
redis.operations.errors - Redis errors
redis.operation.duration - Operation latency
redis.memory.used - Memory usage
redis.memory.usage_percent - Memory usage percentage

Kafka Messages

kafka.messages.total - Total messages
kafka.messages.errors - Message errors
kafka.message.duration - Processing latency
kafka.consumer.lag - Consumer lag per topic

Business Metrics

Bookings

bookings.created - Booking creation count (by status: success, conflict, error)
bookings.conflicts - Conflict detection count
bookings.status_changes - Status transition count
bookings.create.duration - Creation latency histogram

Payments

payments.processed - Payment processing count (by status)
payments.amount - Payment amount histogram

Notifications

notifications.sent - Notification delivery count (by type and status)

Usage Examples

Adding Custom Spans

import { TraceService } from '@football-booking/observability';

@Injectable()
export class MyService {
  constructor(private readonly traceService: TraceService) {}

  async myOperation() {
    return await this.traceService.withSpan(
      'my.operation',
      async (span) => {
        span.setAttribute('operation.id', '123');
        // Your business logic
        return result;
      },
      {
        'operation.type': 'critical',
      },
    );
  }
}

Recording Metrics

import { MetricsService } from '@football-booking/observability';

@Injectable()
export class MyService {
  constructor(private readonly metricsService: MetricsService) {}

  async processPayment(amount: number) {
    const startTime = Date.now();
    try {
      // Process payment
      this.metricsService.recordPayment('success', amount);
      return result;
    } catch (error) {
      this.metricsService.recordPayment('failed');
      throw error;
    } finally {
      const duration = Date.now() - startTime;
      this.metricsService.recordHistogram('payments.processing.duration', duration);
    }
  }
}

Structured Logging

import { LoggerService } from '@football-booking/observability';

@Injectable()
export class MyService {
  constructor(private readonly logger: LoggerService) {}

  async myOperation() {
    this.logger.log('Starting operation', 'MyService', {
      operationId: '123',
      userId: 'user-456',
    });

    try {
      // Business logic
      this.logger.log('Operation completed', 'MyService', { result: 'success' });
    } catch (error) {
      this.logger.error('Operation failed', error.stack, 'MyService', {
        error: error.message,
      });
      throw error;
    }
  }
}

Dashboards

API Gateway Dashboard

Request rate per endpoint
Error rate percentage
Latency (P50, P95, P99)
Active connections
Rate limit hits

Location: config/signoz/dashboards/api-gateway-dashboard.json

Booking Service Dashboard

Booking creation rate
Conflict detection rate
Booking status distribution
Creation latency
Lock acquisition time
Average booking value

Location: config/signoz/dashboards/booking-service-dashboard.json

Database Dashboard

Connection pool usage
Query latency (P95)
Slow queries (> 100ms)
Query error rate
Transaction rate

Location: config/signoz/dashboards/database-dashboard.json

Kafka Dashboard

Consumer lag per topic
Message throughput
Processing latency
Error rate

Location: config/signoz/dashboards/kafka-dashboard.json

Alerts

Error Rate Alerts

High Error Rate: > 1% for 5 minutes (Warning)
Critical Error Rate: > 5% for 2 minutes (Critical)

Latency Alerts

High Latency: P95 > 500ms for 10 minutes (Warning)
Critical Latency: P95 > 1000ms for 5 minutes (Critical)

Business Alerts

Booking Conflicts Spike: Conflict rate > 10% for 5 minutes
Payment Failures: Failure rate > 5% for 5 minutes

Infrastructure Alerts

Kafka Lag: > 10,000 messages for 10 minutes
DB Connections: Usage > 80% for 5 minutes
Redis Memory: Usage > 80% for 5 minutes

Location: config/signoz/alerts/alert-policies.yaml

SLO Monitoring

API Gateway

Availability: 99.9% uptime
Latency: 95% of requests < 200ms

Booking Service

Availability: 99.5% uptime
Success Rate: 95% of bookings succeed
Latency: 90% of bookings < 500ms

Payment Service

Success Rate: 99% of payments succeed
Latency: 95% of payments < 2 seconds

Location: config/signoz/slo-monitoring.yaml

Setup Instructions

1. Environment Variables

Add to .env:

# OpenTelemetry Configuration
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
SERVICE_VERSION=1.0.0
NODE_ENV=development

# Signoz URLs
SIGNOZ_OTEL_COLLECTOR_URL=http://localhost:4318
SIGNOZ_QUERY_SERVICE_URL=http://localhost:8083
SIGNOZ_FRONTEND_URL=http://localhost:3301

2. Enable Observability in Services

Add to `app.module.ts`:

import {
  LoggerModule,
  MetricsInterceptor,
  MetricsModule,
  TraceInterceptor,
  TracingModule,
} from '@football-booking/observability';
import { APP_INTERCEPTOR } from '@nestjs/core';

@Module({
  imports: [
    // ... other modules
    TracingModule,
    MetricsModule,
    LoggerModule,
  ],
  providers: [
    // ... other providers
    {
      provide: APP_INTERCEPTOR,
      useClass: TraceInterceptor,
    },
    {
      provide: APP_INTERCEPTOR,
      useClass: MetricsInterceptor,
    },
  ],
})
export class AppModule {}

Initialize in `main.ts` (before other imports):

import { initializeTracing } from '@football-booking/observability';
initializeTracing('service-name');

3. Start Infrastructure

# Start Signoz and infrastructure
docker compose -f docker-compose.dev.yml up -d signoz-otel-collector signoz-query-service signoz-frontend clickhouse

4. Access Signoz UI

Frontend: http://localhost:3301
Query Service API: http://localhost:8083

5. Import Dashboards

Go to Signoz UI → Dashboards
Click "Import Dashboard"
Upload JSON files from config/signoz/dashboards/

6. Configure Alerts

Go to Signoz UI → Alerts
Import alert policies from config/signoz/alerts/alert-policies.yaml
Configure notification channels (Slack, PagerDuty)

Testing Observability

1. Generate Traffic

# Use k6 for load testing
k6 run scripts/performance-test.js

2. Check Traces

Go to Signoz UI → Traces
Filter by service name
View trace details and spans

3. Check Metrics

Go to Signoz UI → Metrics
Query metrics (e.g., http_requests_total)
View graphs and trends

4. Check Logs

Go to Signoz UI → Logs
Filter by trace ID to correlate with traces
View structured JSON logs

Best Practices

Always initialize tracing first in main.ts before other imports
Use custom spans for critical business operations
Record metrics for all important events
Include context in logs (trace ID, user ID, etc.)
Set appropriate alert thresholds based on SLOs
Monitor error budgets to track SLO compliance
Correlate logs with traces using trace ID