Kubernetes platform engineering

I learned infrastructure by building it, tearing it down and building it again. Each version was right for its moment and wrong when the scope grew.

The five architectures

1. Compose. docker compose up and podman compose up on dedicated Linux servers.
2. systemd services. Units that start at boot and restart on failure, for uptime.
3. AWS ECS with CI/CD. ECR images, IAM, security groups, ACM certificates and Route 53, with the configuration in the repo and the pipeline testing before it deploys.
4. Kubernetes, from scratch. I tore down every ECS service and its pipeline, then worked through Kubernetes the Hard Way and self-hosted a cluster. Along the way I learned certificates, Proxmox, networking and routing, and set up my own DNS server and router at home. That cluster became the blueprint for Tidan Games’ architecture.
5. Seneca Foundation’s official cluster. Everything above, applied to Seneca’s cluster, under the supervision of Harrison, my boss and training mentor during my apprenticeship.

The full story, with dates from git, is in From Podman Compose to systemd services to Kubernetes clusters.

What runs on them

• Delivery: Helm charts, deployed by Argo CD from Git.
• Metrics and logs: Prometheus, Grafana and Loki.
• Data: self-managed PostgreSQL and Redis.
• Secrets and images: OpenBao (the open source Vault fork) and Harbor for Tidan Games; Sealed Secrets and a self-hosted registry for Seneca.
• Edge: ingress with cert-manager certificates.
• Provisioning: Terraform and Ansible.
• Access: RBAC with namespaced service accounts, network policies, certificate SSH, 2FA and host firewalls.

Tidan Games’ cluster is on its own page: Tidan Games Platform.

Where to host it

When Seneca’s cluster needed a long term home, I wrote a cost analysis of AWS and OVH over one year and ten. At the medium tier, OVH is about $780 a year and AWS about $2,041, which is $12,609.60 apart over ten years on one node. See AWS vs OVH costs.

Training cluster

I built and run the k3s cluster where Seneca Foundation’s interns deploy real applications with Helm and GitOps, and I mentor them through it.

Still learning

I’m studying for the Certified Kubernetes Administrator (CKA) exam.