Infrastructure Reading Guide

Read this file before anything else.
Purpose: walk you through the codebase in the right order to build a solid mental model, and tell you exactly where to look when debugging.

---

Part 1 — Sequential Reading Path

Read top to bottom. Each step builds on the previous one.

---

Step 1 — Get the big picture (5 min)

Read: ARCHITECTURE.md — Section 1 (Architecture Overview) and Section 15 (Port Reference)

Questions to answer after this step: - How many nodes are there? What runs on each? - What technology runs the stateful services (DB, cache), and where? - What technology runs the stateless apps (NestJS, FastAPI), and where? - What does traffic pass through before reaching application code?

Mental model after this step:

Browser → nginx-ingress (K3s) → api-gateway pod → service pods
                                                        │
                                              connects to MASTER_IP:PORT
                                                        │
                                             Docker Compose (postgres, redis, kafka...)

---

Step 2 — Understand the environments (3 min)

Read: ../inventory/utm/hosts.ini and ../inventory/utm/group_vars/all.yml

Questions to answer after this step: - What IPs do the three nodes have? - Where is k3s_master_ip used? - How does env_name: "utm-local" affect the deployment?

The most important thing: The k3s_master_ip variable is the backbone of the entire system. It is injected into .env (for Docker Compose) and into K8s Secrets (for pods). Every service that needs to connect outside the K3s cluster uses this IP.

---

Step 3 — Understand the full deploy flow (10 min)

Read: ../playbooks/site.yml — read it entirely, it's only ~30 lines

Questions to answer after this step: - How many steps are there? Which step runs on which node? - Why must Step 6 (stateful stack) run before Step 7 (helm apps)? - Why must Step 5 (kubeconfig) run before Step 7?

Mental model after this step — 8 mandatory sequential steps:

[1] common      → all nodes  → OS preparation
[2] docker      → master     → Docker needed to run Compose
[3] k3s_master  → master     → create cluster, generate node-token
[4] k3s_worker  → workers    → join cluster using token
[5] kubeconfig  → local Mac  → get kubeconfig to run kubectl
[6] stateful    → master     → start DB/cache/kafka BEFORE apps come up
[7] helm_apps   → master     → create K8s Secrets, deploy pods
[8] ingress     → master     → open the door to the outside world

---

Step 4 — Understand stateful infrastructure (15 min)

Read in this order:

1. ../roles/stateful_stack/templates/env.production.j2
   → This is the complete list of secrets the server needs. Read it to know what variables exist.

2. ../secrets/vault.yml
   → Source of truth for every secret. Each vault_xxx here maps to {{ vault_xxx }} in the template above.

3. ../docker-compose/stateful-stack.yml
   → Read the environment: block for each service to understand what config each one receives.
   → Pay attention to the comment explaining why Kafka needs 2 listeners (PLAINTEXT vs EXTERNAL).

4. ../docker-compose/init-scripts/create-multiple-dbs.sh
   → This script runs exactly once during PostgreSQL initialization. Understand it to know how the 7 databases are created.

Questions to answer after this step: - Where does a secret start and where does it end up? (vault.yml → j2 template → .env file → Docker Compose → container) - Why does Kafka need port 9093 instead of 9092? - Where does MASTER_IP in KAFKA_ADVERTISED_LISTENERS come from?

---

Step 5 — Understand observability infrastructure (10 min)

Read: ../docker-compose/observability-stack.yml

Focus on: - clickhouse-init-databases — why is a separate init container needed? - Why are there 2 Redis instances (chatbot-redis on port 6380 and langfuse-redis internal)? - minio — what does LangFuse v3 store in S3? - What does langfuse-web depend on? What is the startup order?

Questions to answer after this step: - What ports does a trace travel through: NestJS pod → OTel collector → ClickHouse? - Why must langfuse-redis use noeviction while chatbot-redis uses allkeys-lru?

---

Step 6 — Understand K3s workloads (10 min)

Read in this order:

1. ../roles/helm_apps/tasks/main.yml
   → Read the backend-secrets and chatbot-secrets creation blocks. This is the complete list of env vars pods receive.

2. ../helm/backend/values.yaml
   → ~20 lines, understand the defaults: replicaCount, image, resources, envFrom.

3. ../helm/backend/templates/deployment.yaml
   → See envFrom: secretRef: backend-secrets — pods pull all config from the K8s Secret.

4. ../helm/chatbot/values.yaml
   → Compare with backend: HPA enabled, different ports (3007/3006).

5. ../helm/ingress/templates/ingress.yaml
   → See ingressClassName: nginx, route / → api-gateway:3000.

Questions to answer after this step: - 6 NestJS services share 1 Helm chart — how are they differentiated? - Why does service discovery between pods use http://user-service:3001 (not an IP)? - What metric does chatbot HPA scale on, and what are the min/max replicas?

---

Step 7 — Understand CI/CD flow (5 min)

Read: 1. ../.github/workflows/README.md — overview of GitHub Actions setup 2. ../.github/workflows/cd-backend.yml — the reusable workflow that actually runs helm 3. ../.github/workflows/_example-caller-backend.yml — how an app repo calls this workflow 4. ../playbooks/deploy_service.yml — manual deploy fallback from local

CI/CD mental model:

git push → GitHub (app repo)
  → GitHub Actions CI: build image → push to Docker Hub
  → GitHub Actions CD: calls reusable workflow from this infra repo
    → SSH into master → helm upgrade --atomic
    → K3s rolling update (auto-rollback on failure)

Key design points: - No dedicated CI server running on a worker node - All deploy logic is centralized in this repo (cd-backend.yml, cd-chatbot.yml) - App repos only call (uses: infra-k3s/.github/workflows/cd-backend.yml@main) - --atomic guarantees automatic rollback if health checks fail

---

Step 8 — Deep-read ARCHITECTURE.md (30 min)

After building your mental model from steps 1–7, read ARCHITECTURE.md from top to bottom.

By now every section will make sense rather than just wash over you.

---

Part 2 — Debug Map: where to look when stuck

Pod won't start

kubectl get pods -n football-booking
kubectl describe pod <pod-name> -n football-booking   # check Events
kubectl logs <pod-name> -n football-booking           # check app logs

Common causes → relevant files:

Symptom	Where to look
ImagePullBackOff	Check image.repository + image.tag in helm/backend/values.yaml or values.utm.yaml
CrashLoopBackOff	Check pod logs. Usually wrong env var → check roles/helm_apps/tasks/main.yml under backend-secrets
Pending (won't schedule)	kubectl describe pod → check node resources → helm/backend/values.yaml resources.requests
Secret not found	kubectl get secret -n football-booking → if missing, helm_apps role hasn't run

---

Service can't connect to database

Check in this order:

# 1. Is the DB container running?
ssh ubuntu@192.168.64.10 "docker ps | grep postgres"

# 2. Is the port open?
ssh ubuntu@192.168.64.10 "ss -tlnp | grep 5432"

# 3. Test connection from inside the pod
kubectl exec -it deployment/user-service -n football-booking -- \
  nc -zv 192.168.64.10 5432

# 4. Check the connection string in the Secret
kubectl get secret backend-secrets -n football-booking -o jsonpath='{.data.DATABASE_URL_USER}' | base64 -d

Files to check: - docker-compose/stateful-stack.yml — port binding, healthcheck - roles/helm_apps/tasks/main.yml — how the connection string is built - inventory/utm/group_vars/all.yml — is k3s_master_ip correct?

---

Kafka not sending/receiving messages

Kafka's dual-listener config is the most common culprit.

# Test from inside a pod
kubectl exec -it deployment/booking-service -n football-booking -- \
  nc -zv 192.168.64.10 9093

# Check Kafka logs
ssh ubuntu@192.168.64.10 \
  "docker logs kafka 2>&1 | grep -i 'listener\|advertised\|error' | tail -30"

# Check MASTER_IP was injected correctly
ssh ubuntu@192.168.64.10 "grep MASTER_IP /home/ubuntu/stacks/football-booking/.env"

Files to check: - docker-compose/stateful-stack.yml — KAFKA_ADVERTISED_LISTENERS must contain EXTERNAL://${MASTER_IP}:9093 - roles/stateful_stack/templates/env.production.j2 — MASTER_IP={{ k3s_master_ip }} - inventory/utm/group_vars/all.yml — is k3s_master_ip correct? - roles/helm_apps/tasks/main.yml — KAFKA_BROKERS="{{ k3s_master_ip }}:9093"

---

Ingress not routing traffic

# 1. Is the nginx-ingress controller running?
kubectl get pods -n ingress-nginx
kubectl get svc -n ingress-nginx   # Does LoadBalancer have an EXTERNAL-IP?

# 2. Has the Ingress resource been created?
kubectl get ingress -n football-booking
kubectl describe ingress football-booking-ingress -n football-booking

# 3. Is api-gateway service healthy?
kubectl get svc api-gateway -n football-booking
kubectl get endpoints api-gateway -n football-booking

Files to check: - roles/ingress/tasks/main.yml — installs nginx-ingress and deploys Ingress - helm/ingress/templates/ingress.yaml — routing rules - helm/ingress/values.utm.yaml — host config

---

LangFuse not working

LangFuse dependency chain:

postgres (langfuse_db) → langfuse-redis → minio → langfuse-web → langfuse-worker

# Check startup status
ssh ubuntu@192.168.64.10 \
  "docker compose -f /home/ubuntu/stacks/football-booking/observability-stack.yml ps"

# Check langfuse-web logs
ssh ubuntu@192.168.64.10 "docker logs langfuse-web 2>&1 | tail -50"

# Test MinIO
curl http://192.168.64.10:9090/minio/health/live

Files to check: - docker-compose/observability-stack.yml — env vars LANGFUSE_S3_EVENT_UPLOAD_*, DATABASE_URL, CLICKHOUSE_URL - roles/stateful_stack/templates/env.production.j2 — MINIO_ROOT_USER, LANGFUSE_INIT_* - secrets/vault.yml — vault_langfuse_*, vault_minio_* variables

Common errors: - LANGFUSE_ENCRYPTION_KEY must be exactly 64 hex characters (openssl rand -hex 32) - LangFuse auto-init only runs once. If public/secret keys change after first init, you must delete the data volume and restart.

---

Chatbot can't call LLM

Flow: chatbot pod → OmniRoute (port 20128) → Groq/Gemini

# 1. Is OmniRoute running?
curl http://192.168.64.10:20128/

# 2. Test from inside the pod
kubectl exec -it deployment/chatbot-service -n football-booking -- \
  curl -s http://192.168.64.10:20128/

# 3. Check chatbot env
kubectl get secret chatbot-secrets -n football-booking \
  -o jsonpath='{.data.OMNIROUTE_BASE_URL}' | base64 -d

Files to check: - roles/helm_apps/tasks/main.yml — LLM_PROVIDERS, OMNIROUTE_BASE_URL - docker-compose/stateful-stack.yml — omni-route service definition

OmniRoute API keys are not stored in files — they are added via the dashboard at http://MASTER_IP:20128.
The omniroute_data volume must still exist. If the volume is lost, API keys must be re-entered.

---

K3s cluster unhealthy

# Check node status
kubectl get nodes -o wide

# Node NotReady → check kubelet
ssh ubuntu@192.168.64.11 "sudo systemctl status k3s-agent"
ssh ubuntu@192.168.64.11 "sudo journalctl -u k3s-agent -n 50"

# Master API not responding
ssh ubuntu@192.168.64.10 "sudo systemctl status k3s"
ssh ubuntu@192.168.64.10 "sudo journalctl -u k3s -n 50"

Files to check: - roles/common/tasks/main.yml — firewall ports, sysctl settings - inventory/utm/group_vars/all.yml — k3s_required_ports - roles/k3s_master/tasks/main.yml — install flags, --tls-san

---

Secret wrong or missing

# List all secrets
kubectl get secrets -n football-booking

# Decode a specific key
kubectl get secret backend-secrets -n football-booking \
  -o jsonpath='{.data.REDIS_URL}' | base64 -d

# Re-create secrets (idempotent)
ansible-playbook -i inventory/utm playbooks/deploy.yml \
  --vault-password-file ~/.vault_pass

Files to check: - secrets/vault.yml — actual secret values - roles/helm_apps/tasks/main.yml — which key maps to which literal value

---

Docker Compose service won't start

# Check status
ssh ubuntu@192.168.64.10 \
  "cd /home/ubuntu/stacks/football-booking && \
   docker compose -f stateful-stack.yml ps && \
   docker compose -f observability-stack.yml ps"

# Check specific service logs
ssh ubuntu@192.168.64.10 \
  "docker logs <container-name> 2>&1 | tail -100"

# Check .env was rendered correctly (careful: file contains secrets)
ssh ubuntu@192.168.64.10 \
  "grep -v PASSWORD /home/ubuntu/stacks/football-booking/.env"

# Re-render .env and restart
ansible-playbook -i inventory/utm playbooks/site.yml \
  --tags stateful_stack --vault-password-file ~/.vault_pass

Files to check: - roles/stateful_stack/templates/env.production.j2 — .env template - secrets/vault.yml — source of truth for secrets - docker-compose/stateful-stack.yml or observability-stack.yml — service config

---

Part 3 — One-Page Mental Model

After reading everything, here is the entire system in one diagram:

SOURCE OF TRUTH
───────────────
secrets/vault.yml                     ← all passwords & API keys live here
inventory/utm/group_vars/all.yml      ← all IPs & versions live here

        │
        │ Ansible renders & copies
        ▼

MASTER NODE (/home/ubuntu/stacks/football-booking/)
────────────────────────────────────────────────────
.env                                  ← vault.yml → j2 template → here
stateful-stack.yml  ──docker compose──► postgres, mongodb, redis,
observability-stack.yml ──────────────► kafka, keycloak, omni-route,
                                        clickhouse, otel-collector,
                                        langfuse, signoz, minio,
                                        chatbot-redis, langfuse-redis

        │ K3s connects via MASTER_IP:PORT
        │
        ▼

K3s CLUSTER (namespace: football-booking)
─────────────────────────────────────────
backend-secrets  ──envFrom──► api-gateway pod      :3000
chatbot-secrets  ──envFrom──► user-service pod     :3001
                              field-service pod     :3002
                              booking-service pod   :3003
                              payment-service pod   :3004
                              notification-service  :3005
                              chatbot-service pod   :3007

        │ ingressClassName: nginx
        │
        ▼

nginx-ingress-controller (namespace: ingress-nginx)
────────────────────────────────────────────────────
/ → api-gateway:3000     ← all external traffic enters here first

---

Part 4 — Debug Session Checklist

Copy-paste and tick each item:

[ ] Identify: which layer is the problem? (ingress / K3s pod / Docker Compose / network / secret)
[ ] kubectl get pods -n football-booking          → are pods Running?
[ ] kubectl get nodes                             → are nodes Ready?
[ ] ssh master: docker ps | grep -c "Up"          → are Compose services Up?
[ ] ssh master: grep MASTER_IP stacks/.../env     → is the IP correct?
[ ] kubectl get secret backend-secrets -n football-booking → does the secret exist?
[ ] curl http://MASTER_IP/health                  → is the API gateway reachable?