Bỏ qua

Deployment Requirements - Football Field Booking System

Version: 1.0 Last Updated: 2026-01-13 Owner: DevOps Lead / Tech Lead Environments: dev (Docker Compose), staging (Kubernetes), production (Kubernetes)

1) Objectives

  • Provide a repeatable, secure deployment process for all services.
  • Ensure environment parity (dev/staging/prod).
  • Support zero/low downtime releases and fast rollbacks.
  • Provide full observability (logs/metrics/traces) in all environments.

2) Infrastructure Requirements

2.1 Compute

  • Kubernetes cluster (managed or self-hosted).
  • Node sizing baseline (MVP):
  • 3 nodes x 2–4 vCPU / 8–16 GB RAM

2.2 Datastores

  • PostgreSQL 15 (managed recommended):
  • user DB
  • field DB
  • booking DB
  • payment DB
  • keycloak DB
  • MongoDB 7 (managed recommended) for notification logs
  • Redis 7 (managed recommended) for cache + locks
  • Kafka (managed recommended) for events

2.3 Observability

  • OpenTelemetry Collector (or Signoz collector)
  • Signoz (or equivalent) for traces/metrics/logs

3) Environments

3.1 Development (Docker Compose)

Required

  • Use docker-compose.dev.yml for local infra
  • Services run via yarn nx serve <service>

Acceptance

  • All infra services healthy (Postgres/Mongo/Redis/Kafka/Keycloak/Signoz)
  • All services health endpoints respond

3.2 Staging (Kubernetes)

Required

  • Mirrors production topology
  • Smaller resources allowed
  • Uses real Keycloak realm/client config (staging)

3.3 Production (Kubernetes)

Required

  • Hardened security settings, TLS, network policies
  • Regular backups, DR testing
  • SLO monitoring + alerting enabled

4) Containerization Requirements

4.1 Docker Images

  • One image per service:
  • api-gateway
  • user-service
  • field-service
  • booking-service
  • payment-service
  • notification-service

Standards

  • Non-root user
  • Minimal base images
  • No secrets baked into images
  • Version tags: service:<semver>-<gitsha> and service:latest (staging only)

4.2 Health Checks

  • Liveness and readiness endpoints per service:
  • GET /health
  • Readiness must check:
  • DB connectivity (where applicable)
  • Kafka connectivity for consumers (optional, with timeouts)

5) Kubernetes Manifests (Reference Templates)

These are templates/specs to implement in IaC (Helm/Kustomize).

5.1 Namespace & Labels

  • Namespace per env: football-booking-dev, football-booking-staging, football-booking-prod
  • Standard labels:
  • app.kubernetes.io/name
  • app.kubernetes.io/part-of=football-booking
  • app.kubernetes.io/component=service
  • app.kubernetes.io/version

5.2 Deployments

Each service deployment must define:

  • replicas (min 2 for gateway in prod)
  • resources (requests/limits)
  • readiness/liveness probes
  • env vars from ConfigMap + Secret
  • rollingUpdate strategy

Example (Skeleton)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
        - name: api-gateway
          image: api-gateway:1.0.0
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 30
            periodSeconds: 20
          resources:
            requests:
              cpu: '200m'
              memory: '256Mi'
            limits:
              cpu: '1'
              memory: '1Gi'

5.3 Services

  • ClusterIP services for internal access.
  • Only expose gateway via Ingress.

5.4 Ingress

Requirements

  • TLS termination
  • Force HTTPS redirect
  • Rate limiting/WAF (recommended)

5.5 HPA

Requirements

  • CPU-based scaling enabled for all stateless services
  • For gateway + booking-service, tune thresholds aggressively

5.6 NetworkPolicies (Prod)

Requirements

  • Allow ingress to gateway only from Ingress controller.
  • Allow gateway → internal services.
  • Restrict services to only required datastores.

6) Configuration & Environment Variables

6.1 Config Sources

  • ConfigMap: non-secret config (ports, feature flags)
  • Secret: passwords, API keys, webhook secrets

6.2 Mandatory Environment Variables (Logical)

Gateway

  • API_GATEWAY_PORT
  • KEYCLOAK_URL
  • KEYCLOAK_REALM
  • KEYCLOAK_CLIENT_ID

Databases

  • DB_*_HOST, DB_*_PORT, DB_*_USERNAME, DB_*_PASSWORD, DB_*_DATABASE

Kafka

  • KAFKA_BROKERS
  • KAFKA_CLIENT_ID
  • KAFKA_SSL_ENABLED (prod)

Redis

  • REDIS_HOST, REDIS_PORT, REDIS_PASSWORD

OpenTelemetry

  • OTEL_EXPORTER_OTLP_ENDPOINT
  • OTEL_SERVICE_NAME

Payment

  • STRIPE_WEBHOOK_SECRET (if used)
  • VNPAY_HASH_SECRET (if used)

Email/ZNS

  • EMAIL_PROVIDER_API_KEY
  • ZALO_ZNS_APP_ID, ZALO_ZNS_SECRET (Phase 2)

7) CI/CD Pipeline Requirements

7.1 CI (on PR)

  • Install deps (yarn)
  • Lint all projects
  • Typecheck/build all services
  • Run tests
  • Run CodeQL/security scanning
  • Build Docker images (optional in PR, required on main)

7.2 CD (on main tag/release)

  • Build & push images to registry
  • Deploy to staging automatically
  • Run smoke tests
  • Manual approval gate for production
  • Deploy prod with rolling update
  • Post-deploy checks (SLO smoke)

8) Deployment Strategies

8.1 Rolling Updates (Default)

Requirements

  • maxUnavailable=0 for gateway
  • readiness probes required
  • rollback if error rate/latency spikes

8.2 Blue/Green (Optional)

Used for risky migrations or gateway changes.

9) Database Migrations

Requirements

  • Migrations must run automatically during deployment:
  • as initContainer or pre-deploy job
  • Must be backward compatible for rolling deploy:
  • expand → migrate → contract strategy
  • Separate migration job per service DB

10) Backups & Disaster Recovery

10.1 Backup Policy

  • Daily full backups (Postgres)
  • Hourly incremental (if available)
  • Retention: 30 days

10.2 DR Testing

  • Weekly restore test in staging for at least one DB
  • Quarterly full DR exercise

11) Monitoring & Alerting Setup

Requirements

  • OpenTelemetry instrumentation enabled in all services
  • Dashboards:
  • RED metrics per endpoint
  • DB saturation metrics
  • Kafka consumer lag
  • Redis latency/memory
  • Alerts:
  • error rate spikes
  • p95 latency breaches
  • Kafka lag thresholds
  • DB connection pool saturation

12) Operations Runbook (Minimum)

12.1 Standard Checks

  • Service health endpoints
  • Logs/traces for failing requestId/traceId
  • Kafka lag dashboards
  • DB slow queries and pool usage

12.2 Common Incidents

  • Redis unavailable: reduce booking throughput, enable fallback, scale DB.
  • Kafka unavailable: degrade notifications, enable outbox replay (if implemented).
  • Keycloak unavailable: protected endpoints return 503; prioritize recovery.

Document Version: 1.0 Last Updated: 2026-01-13 Owner: DevOps Lead / Tech Lead