Football Booking — Infrastructure Architecture

Purpose: This document describes the complete K3s infrastructure architecture of the Football Booking system, covering the role of each file, deployment flow, and how components interconnect. Designed to deploy on 3 local UTM VMs (development) or 3 Oracle Cloud VMs (production-equivalent).

1. Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                         INTERNET / Local Browser                        │
└─────────────────────────────┬───────────────────────────────────────────┘
                              │ HTTP :80 / :443
                              ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  MASTER NODE  (192.168.64.10 / Oracle VM)                               │
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │  nginx-ingress-controller  (K3s, namespace: ingress-nginx)       │   │
│  │  ServiceLB klipper-lb → auto-assigns node IP                     │   │
│  └───────────────────────┬──────────────────────────────────────────┘   │
│                          │  routes /  → api-gateway:3000                │
│  ┌───────────────────────▼──────────────────────────────────────────┐   │
│  │  K3s Pods  (namespace: football-booking)                         │   │
│  │                                                                   │   │
│  │  api-gateway:3000 ──► user-service:3001                         │   │
│  │                   ──► field-service:3002                         │   │
│  │                   ──► booking-service:3003                       │   │
│  │                   ──► payment-service:3004                       │   │
│  │                   ──► notification-service:3005                  │   │
│  │                   ──► chatbot-service:3007                       │   │
│  └───────────────────────┬──────────────────────────────────────────┘   │
│                          │  MASTER_IP:PORT                              │
│  ┌───────────────────────▼──────────────────────────────────────────┐   │
│  │  Docker Compose — Stateful Stack                                 │   │
│  │                                                                   │   │
│  │  postgres:5432   mongodb:27017   redis:6379   kafka:9093         │   │
│  │  keycloak:8080   omni-route:20128                                │   │
│  └──────────────────────────────────────────────────────────────────┘   │
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │  Docker Compose — Observability Stack                            │   │
│  │                                                                   │   │
│  │  clickhouse:8123/9000   otel-collector:4317/4318                 │   │
│  │  signoz-frontend:3301   langfuse-web:3100                        │   │
│  │  chatbot-redis:6380     langfuse-minio:9090                      │   │
│  └──────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────┘

WORKER NODE 1  (192.168.64.11)   ──► runs K3s workloads (pods)
WORKER NODE 2  (192.168.64.12)   ──► runs K3s workloads (pods)

Separation of Concerns

Layer	Technology	Node	Reason
Stateful services (DB, cache, queue, auth)	Docker Compose	Master	Need persistent data, not horizontally scalable
Stateless apps (NestJS, FastAPI)	K3s + Helm	Workers	Horizontal scaling, rolling deploy, auto health-check
Observability (SigNoz, LangFuse)	Docker Compose	Master	Separate stack, start/stop independently
CI/CD (GitHub Actions)	Managed cloud	N/A	Stateless reusable workflows, no dedicated server needed

2. Directory Structure

football-booking-infra-k3s/
│
├── ansible.cfg                    # Ansible global config
├── README.md                      # Project overview
├── docs/                          # Architecture & guides (this folder)
│   ├── ARCHITECTURE.md            # This file (English)
│   ├── ARCHITECTURE.vi.md         # Vietnamese version
│   ├── READING_GUIDE.md           # Sequential reading guide (English)
│   └── READING_GUIDE.vi.md        # Vietnamese version
│
├── inventory/                     # Environment definitions
│   ├── utm/                       # Local VMs via UTM (development)
│   │   ├── hosts.ini              # IPs: 192.168.64.10/11/12
│   │   └── group_vars/
│   │       ├── all.yml            # Shared vars: k3s_master_ip, versions, env_name
│   │       ├── k3s_master.yml     # Master-node-specific vars
│   │       └── k3s_workers.yml    # Worker-node-specific vars
│   └── oracle/                    # Oracle Cloud VMs (production-equivalent)
│       ├── hosts.ini              # Oracle instance public IPs
│       └── group_vars/
│           └── all.yml            # Same as utm but env_name: "oracle"
│
├── playbooks/                     # Entry points — run these
│   ├── site.yml                   # Full cluster setup (first run or rebuild)
│   ├── deploy.yml                 # Re-deploy all app services
│   └── deploy_service.yml         # Re-deploy a single service (manual fallback)
│
├── roles/                         # Ansible roles — one per setup step
│   ├── common/                    # OS baseline for all nodes
│   ├── docker/                    # Install Docker CE on master
│   ├── k3s_master/                # Install K3s server on master
│   ├── k3s_worker/                # Join K3s cluster from workers
│   ├── kubeconfig/                # Fetch kubeconfig to local Mac
│   ├── stateful_stack/            # Deploy Docker Compose stateful stack
│   ├── helm_apps/                 # Create K8s Secrets + deploy Helm charts
│   └── ingress/                   # Install nginx-ingress + deploy Ingress resource
│
├── helm/                          # Helm charts for K3s workloads
│   ├── backend/                   # Shared chart for 6 NestJS microservices
│   ├── chatbot/                   # Dedicated chart for FastAPI chatbot
│   └── ingress/                   # Chart defining Ingress routing rules
│
├── docker-compose/                # Docker Compose for stateful services on master
│   ├── stateful-stack.yml         # Core infra: PG, Mongo, Redis, Kafka, Keycloak, OmniRoute
│   ├── observability-stack.yml    # Monitoring: ClickHouse, SigNoz, LangFuse, MinIO
│   ├── config/
│   │   └── otel-collector-config.yaml  # OpenTelemetry Collector pipeline config
│   └── init-scripts/
│       ├── create-multiple-dbs.sh      # Create 7 PG databases + chatbot user on init
│       └── mongo-init.js               # Create notification_db collections on init
│
├── .github/workflows/             # GitHub Actions CI/CD
│   ├── cd-backend.yml             # Reusable: deploy 1 backend service via SSH + helm
│   ├── cd-chatbot.yml             # Reusable: deploy chatbot service
│   ├── _example-caller-backend.yml # Template to copy into backend app repo
│   ├── _example-caller-chatbot.yml # Template to copy into chatbot app repo
│   └── README.md                  # GitHub Actions setup guide
│
├── secrets/
│   ├── vault.yml                  # All secrets (MUST encrypt before commit)
│   └── .gitignore                 # Prevents committing plaintext vault
│
└── scripts/
    ├── check-before-commit.sh     # Security check before git commit
    ├── setup-utm-network.sh       # Configure UTM bridged/shared network
    └── verify-cluster.sh          # Check cluster health after deploy

3. Inventory — Environment Management

`inventory/utm/hosts.ini`

Defines 3 UTM VMs with static IPs:

[k3s_master]  192.168.64.10
[k3s_workers] 192.168.64.11, 192.168.64.12

SSH key: ~/.ssh/utm_k3s

`inventory/oracle/hosts.ini`

Same structure but uses Oracle Cloud public IPs and key ~/.ssh/oracle_football_booking.

`inventory/utm/group_vars/all.yml` — Key Variables

Variable	Value	Used in
`k3s_master_ip`	`192.168.64.10`	Injected into `.env` → Kafka EXTERNAL, K8s Secrets
`k3s_version`	`v1.29.4+k3s1`	Install exact K3s version
`env_name`	`utm-local`	Determines values file: `utm` or `oracle`
`backend_image_prefix`	`sangtrandev00`	Docker Hub registry prefix

The `env_short` mechanism

Roles read env_name to select the correct Helm values file:

env_short: "{{ 'utm' if 'utm' in env_name else 'oracle' }}"
# → "utm"    when using inventory/utm
# → "oracle" when using inventory/oracle

Helm then deploys with: -f values.yaml -f values.{{ env_short }}.yaml

4. Playbooks — Entry Points

`playbooks/site.yml` — Full cluster setup

Run once when creating new VMs or rebuilding from scratch. Executes 8 sequential steps:

Step 1: common         → all nodes    → OS baseline, swap, firewall, sysctl
Step 2: docker         → master only  → Install Docker CE + Compose plugin
Step 3: k3s_master     → master only  → Install K3s server, disable Traefik
Step 4: k3s_worker     → workers      → Join K3s cluster with node-token
Step 5: kubeconfig     → master→local → Copy ~/.kube/football-k3s.yaml to Mac
Step 6: stateful_stack → master only  → Render .env, copy files, docker compose up
Step 7: helm_apps      → master only  → Create K8s Secrets, deploy Helm charts
Step 8: ingress        → master only  → Install nginx-ingress, deploy Ingress rules

ansible-playbook -i inventory/utm playbooks/site.yml --vault-password-file ~/.vault_pass

`playbooks/deploy.yml` — Re-deploy all apps

Use when rolling out new images for all services:

ansible-playbook -i inventory/utm playbooks/deploy.yml \
  -e "backend_image_tag=abc1234" -e "chatbot_image_tag=abc1234" \
  --vault-password-file ~/.vault_pass

Only runs the helm_apps role — updates Helm releases with the new tag.

`playbooks/deploy_service.yml` — Re-deploy a single service

Used for manual deploys or as a GitHub Actions fallback:

ansible-playbook -i inventory/utm playbooks/deploy_service.yml \
  -e "service=user-service" -e "image_tag=abc1234"

Runs helm upgrade --install for exactly one service then verifies the rollout.

5. Roles — Step-by-Step Details

`roles/common/` — OS baseline

Runs on: all nodes
What it does: - Update apt, install base packages (curl, wget, ufw, htop) - Configure sysctl for K3s: net.ipv4.ip_forward=1, bridge-nf-call-iptables=1 - Create 2GB swap file (K3s needs RAM headroom) - Open firewall ports: 22 (SSH), 6443 (K3s API), 8472 (Flannel VXLAN), 10250 (Kubelet)

Key file: templates/sysctl-k3s.conf.j2 → /etc/sysctl.d/99-k3s.conf

`roles/docker/` — Docker on master

Runs on: master only
What it does: Install Docker CE + Compose plugin, add user ubuntu to docker group.

`roles/k3s_master/` — K3s control plane

Runs on: master only
What it does:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.4+k3s1 sh -s - server \
  --tls-san 192.168.64.10 \
  --disable traefik          # ← Use nginx-ingress instead

Then reads /var/lib/rancher/k3s/server/node-token and sets fact k3s_token for workers.

Why disable Traefik? K3s bundles Traefik by default, but this project uses nginx-ingress for more flexible configuration (proxy-body-size, timeouts, path rewriting).

`roles/k3s_worker/` — Join cluster

Runs on: workers
What it does:

curl -sfL https://get.k3s.io | K3S_URL=https://192.168.64.10:6443 \
  K3S_TOKEN={{ k3s_token }} sh -

`roles/kubeconfig/` — Fetch kubeconfig

Runs on: master → copies to local Mac
What it does: Copies /etc/rancher/k3s/k3s.yaml to ~/.kube/football-k3s.yaml, replacing 127.0.0.1 with k3s_master_ip for remote access.

`roles/stateful_stack/` — Docker Compose infrastructure

Runs on: master only
Execution order (tasks/main.yml): 1. Create directory /home/ubuntu/stacks/football-booking/ 2. Render .env from Ansible Vault → /home/ubuntu/stacks/football-booking/.env 3. Copy stateful-stack.yml, observability-stack.yml 4. Copy init-scripts/ (PostgreSQL + MongoDB init) 5. Copy config/otel-collector-config.yaml 6. docker compose up -d for stateful-stack 7. docker compose up -d for observability-stack

Key files: - templates/env.production.j2 → Jinja2 template renders Vault secrets into the .env file on the server - vars/vault.yml → Local copy of vault vars (used when running this role standalone)

`roles/helm_apps/` — Deploy K8s workloads

Runs on: master only
Execution order (tasks/main.yml): 1. Install Helm if not present 2. Create namespace football-booking 3. Copy the entire helm/ directory to /tmp/helm/ 4. Create K8s Secret backend-secrets (30+ env vars for NestJS services) 5. Create K8s Secret chatbot-secrets (LLM keys, Redis, DB for chatbot) 6. Deploy chatbot: helm upgrade --install chatbot-service /tmp/helm/chatbot ... 7. Deploy backend: loop over 6 services with helm upgrade --install

Why --dry-run=client -o yaml | kubectl apply -f -?
To make the operation idempotent — running multiple times won't fail with "already exists". kubectl create secret cannot run twice for the same resource.

`roles/ingress/` — Nginx Ingress Controller

Runs on: master only
What it does: 1. Set env_short fact 2. Add Helm repo ingress-nginx 3. helm upgrade --install ingress-nginx with service.type=LoadBalancer 4. K3s ServiceLB (klipper-lb) automatically assigns the node IP to the LoadBalancer service 5. Deploy football-ingress Helm chart → creates Ingress resource routing / → api-gateway:3000

6. Docker Compose Stacks

`docker-compose/stateful-stack.yml` — Core Infrastructure

Service	Image	Port (host)	Role
`postgres`	`postgres:16-alpine`	`5432`	7 databases (user, field, booking, payment, keycloak, langfuse, chatbot)
`mongodb`	`mongo:7`	`27017`	notification_db for notification-service
`redis`	`redis:7-alpine`	`6379`	Cache, rate limiting, sessions (allkeys-lru)
`kafka`	`apache/kafka:4.3.1`	`9093` (EXTERNAL)	Event bus between microservices — KRaft mode (no Zookeeper)
`keycloak`	`keycloak:24.0.1`	`8080`	OAuth2/OIDC Identity Provider
`omni-route`	`diegosouzapw/omniroute`	`20128`	Multi-provider LLM router (Groq, Gemini)

Kafka dual listener — why is it needed?
K3s pods on worker nodes cannot resolve the hostname kafka (Docker DNS only works within the Docker network). Two listeners are required:

PLAINTEXT://kafka:9092       → used by Docker containers internally
EXTERNAL://MASTER_IP:9093    → used by K3s pods connecting via host IP

MASTER_IP is injected from .env (rendered by Ansible from k3s_master_ip).

Port bindings on 0.0.0.0:
All services bind to 0.0.0.0 (not 127.0.0.1) so K3s pods on worker nodes can reach them via MASTER_IP:PORT. In production, restrict access with firewall/VPC rules.

`docker-compose/observability-stack.yml` — Monitoring & LLM Observability

Service	Image	Port (host)	Role
`clickhouse`	`clickhouse:24.8-alpine`	`8123`, `9000`	OLAP store for SigNoz traces/metrics + LangFuse analytics
`clickhouse-init-databases`	same	-	One-time init: creates `signoz_metrics`, `signoz_traces`, `signoz_logs`
`otel-collector`	`signoz-otel-collector`	`4317`, `4318`	Receives OTLP traces from K3s pods, writes to ClickHouse
`query-service`	`signoz/query-service`	internal	SigNoz backend API
`signoz-frontend`	`signoz/frontend`	`3301`	SigNoz dashboard UI
`chatbot-redis`	`redis:7-alpine`	`6380`	Dedicated session store for chatbot (allkeys-lru)
`langfuse-redis`	`redis:7-alpine`	internal	LangFuse ingestion queue (noeviction)
`minio`	`cgr.dev/chainguard/minio`	`9090`, `9091`	S3 blob store for LangFuse trace attachments
`langfuse-web`	`langfuse/langfuse:3`	`3100`	LangFuse UI + API
`langfuse-worker`	`langfuse/langfuse-worker:3`	internal	Async processor for LangFuse traces

Why 3 separate Redis instances? - redis (6379, stateful-stack): Backend services, allkeys-lru eviction - chatbot-redis (6380, observability-stack): Chatbot session store, allkeys-lru - langfuse-redis (internal): LangFuse ingestion queue, must use noeviction to prevent trace data loss

clickhouse-init-databases:
SigNoz requires 3 separate databases. This container runs clickhouse-client --query="CREATE DATABASE IF NOT EXISTS ..." then exits. Both otel-collector and query-service depend on service_completed_successfully.

`docker-compose/init-scripts/`

create-multiple-dbs.sh — Runs once during PostgreSQL initialization: 1. Reads POSTGRES_MULTIPLE_DATABASES=user_db,field_db,...,chatbot_db 2. Creates each database with psql CREATE DATABASE 3. Creates dedicated user chatbot with password from CHATBOT_DB_PASSWORD 4. Grants CONNECT + schema permissions on chatbot_db to user chatbot

Why a dedicated chatbot user? Least-privilege — chatbot only needs read/write access to chatbot_db, not admin access. Mirrors the production setup.

mongo-init.js — Runs once during MongoDB initialization:
Creates notification_db, collections notifications + email_logs, and required indexes.

7. Helm Charts — K3s Workloads

`helm/backend/` — NestJS Microservices

Shared across 6 services: api-gateway, user-service, field-service, booking-service, payment-service, notification-service.

Chart.yaml              → metadata: name=backend, version=0.1.0
values.yaml             → defaults: replicaCount=1, image, resources, HPA disabled
values.utm.yaml         → UTM overrides (lower resource limits)
values.oracle.yaml      → Oracle overrides (higher limits, HPA enabled)
templates/
  deployment.yaml       → K8s Deployment, envFrom: backend-secrets
  service.yaml          → ClusterIP service
  hpa.yaml              → HorizontalPodAutoscaler (optional)
  _helpers.tpl          → Template helpers: fullname, labels

Each service deploys as a separate Helm release:

helm upgrade --install user-service /tmp/helm/backend \
  --set image.repository=sangtrandev00/user-service \
  --set image.tag=abc1234

Env vars: All injected via envFrom: [{secretRef: {name: backend-secrets}}].
No hard-coded env vars in pods — everything comes from the backend-secrets K8s Secret.

`helm/chatbot/` — FastAPI Chatbot

Similar to backend but: - Port: 3007 (service) → 3006 (container) - HPA: enabled by default, scales 1→5 replicas at 70% CPU - Secret: chatbot-secrets (LLM keys, Pinecone, Redis 6380, LangFuse)

`helm/ingress/` — Routing Rules

values.yaml        → host: "" (IP-based), apiGateway.service=api-gateway, port=3000
values.utm.yaml    → host: "" (direct IP access)
values.oracle.yaml → host: "" (can set a domain if available)
templates/
  ingress.yaml     → Ingress resource with ingressClassName: nginx

Traffic flow after deploy:

Request → nginx-ingress-controller → api-gateway:3000 → (K8s internal DNS) → services

8. Secrets Management

Two vault files structure

secrets/vault.yml                    ← MAIN file, loaded by site.yml/deploy.yml
roles/stateful_stack/vars/vault.yml  ← Subset copy, loaded when running this role standalone

Both files must have the same set of variables. When adding a new secret, update both.

`secrets/vault.yml` — All secrets

Group	Variables
PostgreSQL	`vault_db_admin_password`
Redis main	`vault_redis_password`
Chatbot Redis	`vault_chatbot_redis_password`
Chatbot DB user	`vault_chatbot_db_password`
Kafka UI	`vault_kafka_ui_password`
ClickHouse	`vault_clickhouse_password`
Keycloak	`vault_keycloak_admin_password`, `vault_keycloak_client_secret`
Backend secrets	`vault_internal_service_secret`, `vault_jwt_secret`, `vault_jwt_encryption_key`
MinIO	`vault_minio_root_user`, `vault_minio_root_password`
LangFuse	`vault_langfuse_nextauth_secret`, `vault_langfuse_salt`, `vault_langfuse_encryption_key`, `vault_langfuse_redis_auth`, `vault_langfuse_init_email`, `vault_langfuse_password`, `vault_langfuse_public_key`, `vault_langfuse_secret_key`
LLM	`vault_groq_api_key`, `vault_pinecone_api_key`, `vault_pinecone_index_name`
GitHub Actions	`vault_gh_deploy_key_comment`

Secrets flow to server

secrets/vault.yml (encrypted)
        │  ansible-vault decrypt (at runtime)
        ▼
roles/stateful_stack/templates/env.production.j2
        │  Ansible template render (Jinja2)
        ▼
/home/ubuntu/stacks/football-booking/.env  (mode 0600, on server)
        │  Docker Compose ${VAR} expansion
        ▼
Environment variables inside containers

secrets/vault.yml (encrypted)
        │  ansible-vault decrypt (at runtime)
        ▼
roles/helm_apps/tasks/main.yml
        │  kubectl create secret --from-literal=KEY=VALUE
        ▼
K8s Secret: backend-secrets  (namespace: football-booking)
K8s Secret: chatbot-secrets  (namespace: football-booking)
        │  envFrom: secretRef
        ▼
Environment variables inside K3s pods

Ansible Vault commands

# Encrypt before committing
ansible-vault encrypt secrets/vault.yml

# Edit encrypted file
ansible-vault edit secrets/vault.yml

# Run playbook with vault
ansible-playbook -i inventory/utm playbooks/site.yml \
  --vault-password-file ~/.vault_pass

9. K8s Secrets — Connection Strings

`backend-secrets` — for 6 NestJS microservices

Key	Example value	Used by
`DB_HOST`	`192.168.64.10`	All services
`DATABASE_URL_USER`	`postgresql://football_admin:pw@MASTER_IP:5432/user_db`	user-service
`MONGODB_URI`	`mongodb://football_admin:pw@MASTER_IP:27017/notification_db`	notification-service
`REDIS_URL`	`redis://:pw@MASTER_IP:6379`	All services
`KAFKA_BROKERS`	`MASTER_IP:9093`	field, booking, payment, notification
`KEYCLOAK_URL`	`http://MASTER_IP:8080`	api-gateway, user-service
`USER_SERVICE_URL`	`http://user-service:3001`	api-gateway (K8s DNS)
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://MASTER_IP:4318`	All services
`INTERNAL_SERVICE_SECRET`	random 32-char	Header `X-Internal-Service`

`chatbot-secrets` — for FastAPI chatbot

Key	Example value	Note
`LLM_PROVIDERS`	`omniroute`	Uses OmniRoute router
`OMNIROUTE_BASE_URL`	`http://MASTER_IP:20128/v1`	API keys configured via OmniRoute dashboard
`GROQ_API_KEY`	`gsk_...`	Fallback / add to OmniRoute
`REDIS_URL`	`redis://:pw@MASTER_IP:6380`	Dedicated chatbot-redis (port 6380)
`CHATBOT_DB_URL`	`postgresql://chatbot:pw@MASTER_IP:5432/chatbot_db`	Dedicated `chatbot` user
`LANGFUSE_HOST`	`http://MASTER_IP:3100`	LLM observability
`LANGFUSE_PUBLIC_KEY`	`pk-lf-football-prod`	Must match `LANGFUSE_INIT_PROJECT_PUBLIC_KEY`
`PINECONE_API_KEY`	`pcsk_...`	Vector DB for RAG

10. Network Communication

┌─────────────────────────────────────────────────────────┐
│  football-booking-network  (Docker bridge)              │
│                                                          │
│  postgres   mongodb   redis   kafka   keycloak          │
│  omni-route   clickhouse   otel-collector               │
│  langfuse-web   langfuse-worker   minio                 │
│  chatbot-redis   langfuse-redis                         │
│  signoz-frontend   query-service                        │
└──────────────────────────┬──────────────────────────────┘
                           │ All containers in the same
                           │ Docker network use hostnames
                           │ (postgres:5432, kafka:9092, ...)
                           │
┌──────────────────────────▼──────────────────────────────┐
│  K3s Pods (worker nodes)                                │
│                                                          │
│  Connect via MASTER_IP:PORT (host binding 0.0.0.0)     │
│  MASTER_IP:5432    → postgres                           │
│  MASTER_IP:27017   → mongodb                            │
│  MASTER_IP:6379    → redis                              │
│  MASTER_IP:6380    → chatbot-redis                      │
│  MASTER_IP:9093    → kafka EXTERNAL listener            │
│  MASTER_IP:8080    → keycloak                           │
│  MASTER_IP:4318    → otel-collector HTTP                │
│  MASTER_IP:3100    → langfuse-web                       │
│  MASTER_IP:20128   → omni-route LLM router              │
│                                                          │
│  K8s internal DNS (ClusterIP):                          │
│  api-gateway → user-service:3001 (K8s DNS)              │
│  api-gateway → chatbot-service:3007 (K8s DNS)           │
└─────────────────────────────────────────────────────────┘

11. OmniRoute — LLM Router

Role: Multi-provider LLM router. The chatbot does not call Groq/Gemini directly — it calls OmniRoute, which routes to the appropriate provider.

chatbot K3s pod
  │  POST http://MASTER_IP:20128/v1/chat/completions
  │  model: "football-fast" or "football-power"
  ▼
omni-route:20128 (Docker Compose, master node)
  │  Routes based on configured model aliases
  ├─► Groq     (llama-3.1-8b-instant)
  ├─► Gemini   (gemini-2.0-flash)
  └─► Anthropic (claude-haiku-4-5)

API keys are added via the OmniRoute dashboard (http://MASTER_IP:20128), not hard-coded in env. Stored in the omniroute_data volume.

12. CI/CD Flow (GitHub Actions)

Developer pushes → GitHub main branch (app repo)
        │
        ▼
GitHub Actions — CI job (runs on GitHub cloud)
  │  dorny/paths-filter → detect which service changed
  │  docker/build-push-action → build image
  │  Push sangtrandev00/<service>:<git-sha> → Docker Hub
        │
        ▼
GitHub Actions — CD job (calls reusable workflow from this infra repo)
  uses: .../football-booking-infra-k3s/.github/workflows/cd-backend.yml@main
        │
        ▼
appleboy/ssh-action → SSH into master node
  helm upgrade --install <service> /tmp/helm/backend \
    --set image.tag=<git-sha> --atomic
        │
        ▼
K3s rolling update → zero downtime
  old pod running → new pod starts → health check passes → old pod terminates
  (--atomic: auto-rollback if health check fails)

Reusable workflows live in this repo: - .github/workflows/cd-backend.yml — deploys 1 backend service - .github/workflows/cd-chatbot.yml — deploys chatbot service

App repos do not contain deploy logic — they only call workflows from this repo, keeping all deploy logic centralized.

Manual deploy (fallback without GitHub Actions):

ansible-playbook -i inventory/oracle playbooks/deploy_service.yml \
  -e "service=user-service" -e "image_tag=abc1234"

GitHub Secrets required in each app repo: - DEPLOY_SSH_KEY — SSH private key for master node - K3S_MASTER_HOST — master node IP - DOCKER_HUB_USERNAME + DOCKER_HUB_TOKEN

Full setup guide: .github/workflows/README.md

13. First Deploy Guide (UTM)

Step 1: Preparation

# Create 3 UTM VMs (Ubuntu 22.04), configure static IPs:
# Master: 192.168.64.10
# Worker1: 192.168.64.11
# Worker2: 192.168.64.12

# Create SSH key
ssh-keygen -t ed25519 -f ~/.ssh/utm_k3s

# Copy public key to each VM
ssh-copy-id -i ~/.ssh/utm_k3s.pub ubuntu@192.168.64.10
ssh-copy-id -i ~/.ssh/utm_k3s.pub ubuntu@192.168.64.11
ssh-copy-id -i ~/.ssh/utm_k3s.pub ubuntu@192.168.64.12

Step 2: Fill in secrets

# Edit vault.yml, replace all "CHANGE_ME" with real values

# Generate required random values:
openssl rand -hex 32    # → LANGFUSE_ENCRYPTION_KEY (must be 64 hex chars)
openssl rand -base64 32 # → JWT_SECRET, NEXTAUTH_SECRET, etc.

# Encrypt
ansible-vault encrypt secrets/vault.yml
ansible-vault encrypt roles/stateful_stack/vars/vault.yml
echo "my-vault-password" > ~/.vault_pass
chmod 600 ~/.vault_pass

Step 3: Deploy

# Full cluster setup (~15-20 minutes)
ansible-playbook -i inventory/utm playbooks/site.yml \
  --vault-password-file ~/.vault_pass

# Verify
./scripts/verify-cluster.sh utm
kubectl --kubeconfig ~/.kube/football-k3s.yaml get pods -n football-booking

Step 4: Configure OmniRoute

# Open dashboard
open http://192.168.64.10:20128

# Add API keys via the UI:
# - Groq: gsk_xxx → alias "football-fast" (llama-3.1-8b-instant)
# - Gemini: AIzaxxx → alias "football-power"

Step 5: Verify services

# Test API Gateway
curl http://192.168.64.10/health

# LangFuse UI
open http://192.168.64.10:3100

# SigNoz APM
open http://192.168.64.10:3301

# Keycloak Admin
open http://192.168.64.10:8080

14. Day-to-Day Operations

# Re-deploy after code push
ansible-playbook -i inventory/utm playbooks/deploy.yml \
  -e "backend_image_tag=$(git rev-parse --short HEAD)"

# Deploy a specific service
ansible-playbook -i inventory/utm playbooks/deploy_service.yml \
  -e "service=user-service" -e "image_tag=abc1234"

# Stream K3s pod logs
kubectl --kubeconfig ~/.kube/football-k3s.yaml logs \
  -n football-booking deployment/user-service -f

# Stream Docker Compose logs on master
ssh ubuntu@192.168.64.10 \
  "cd /home/ubuntu/stacks/football-booking && docker compose -f stateful-stack.yml logs kafka -f"

# Scale chatbot
kubectl --kubeconfig ~/.kube/football-k3s.yaml scale \
  deployment/chatbot-service --replicas=3 -n football-booking

# Restart a service
kubectl --kubeconfig ~/.kube/football-k3s.yaml rollout restart \
  deployment/api-gateway -n football-booking

15. Quick Port Reference

Port	Service	Accessible from
80/443	nginx-ingress	Internet / Browser
3000	api-gateway (K8s ClusterIP)	Internal K8s only
3100	langfuse-web	Browser
3301	signoz-frontend	Browser
4317	otel-collector gRPC	K3s pods
4318	otel-collector HTTP	K3s pods
5432	PostgreSQL	K3s pods, local dev
6379	Redis (backend)	K3s pods
6380	chatbot-redis	K3s pods (chatbot only)
8080	Keycloak	K3s pods, browser
9093	Kafka EXTERNAL	K3s pods
20128	OmniRoute LLM router	Browser, K3s pods
27017	MongoDB	K3s pods

Football Booking — Infrastructure Architecture

1. Architecture Overview

Separation of Concerns

2. Directory Structure

3. Inventory — Environment Management

inventory/utm/hosts.ini

inventory/oracle/hosts.ini

inventory/utm/group_vars/all.yml — Key Variables

The env_short mechanism

4. Playbooks — Entry Points

playbooks/site.yml — Full cluster setup

playbooks/deploy.yml — Re-deploy all apps

playbooks/deploy_service.yml — Re-deploy a single service