Introduction

The Artemis Cluster! :octocat:

... managed with Flux, Renovate, and GitHub Actions 🤖

📖 Overview

This is a mono repository for my home infrastructure and Kubernetes cluster. I try to adhere to Infrastructure as Code (IaC) and GitOps practices using tools like Kubernetes, Flux, Renovate, and GitHub Actions.

⛵ Kubernetes

Core Components

  • sops: Managed secrets for Kubernetes and Terraform which are commited to Git.

Directories

This Git repository contains the following directories under Kubernetes.

📁 kubernetes
├── 📁 apps           # applications
├── 📁 bootstrap      # bootstrap procedures
├── 📁 flux           # core flux configuration
└── 📁 ...             # other clusters

K3S Node Prep

Create cloudinit for repeatable + consistent nodes.

sudo qm create 8000 --name "ubuntu-cloudinit" --ostype l26 \
    --memory 1024 \
    --agent 1 \
    --bios ovmf --machine q35 --efidisk0 local-ceph-01:0,pre-enrolled-keys=0 \
    --cpu host --socket 1 --cores 1 \
    --vga serial0 --serial0 socket  \
    --net0 virtio,bridge=vmbr0

sudo qm importdisk 8000 mantic-server-cloudimg-amd64.img local-ceph-01
sudo qm set 8000 --scsihw virtio-scsi-pci --virtio0 local-ceph-01:vm-8000-disk-1,discard=on
sudo qm set 8000 --boot order=virtio0
sudo qm set 8000 --ide2 local-ceph-01:cloudinit

sudo qm set 8000 --cicustom "vendor=local:snippets/vendor.yaml"
sudo qm set 8000 --tags template,23.10,cloudinit,ubuntu
sudo qm set 8000 --ciuser root
sudo qm set 8000 --cipassword $(openssl passwd -6 $CLEARTEXT_PASSWORD)
sudo qm set 8000 --sshkeys ~/.ssh/authorized_keys
sudo qm set 8000 --ipconfig0 ip=dhcp

Configure 5 new nodes

qm clone 8000  201 --name k3s-02 --full
qm clone 8000  202 --name k3s-02 --full
qm clone 8000  203 --name k3s-03 --full
qm clone 8000  204 --name k3s-04 --full
qm clone 8000  205 --name k3s-05 --full

qm resize 201 virtio0 +12.5G
qm resize 202 virtio0 +12.5G
qm resize 203 virtio0 +12.5G
qm resize 204 virtio0 +12.5G
qm resize 205 virtio0 +12.5G

Created a cloud init drive and set some parametes so we could clone the drives. Gave each node 16GB

NOTE

On one of the nodes there was issues pinging so ran the following:

cd /var/lib/dpkg/info/ && sudo apt install --reinstall $(grep -l 'setcap' * | sed -e 's/\.[^.]*$//g' | sort --unique)

Talos Machine Prep

brew install talosctl

brew install kubectl

brew install helm

brew install helmfile

brew install fluxcd/tap/flux

sops --decrypt {{.KUBERNETES_DIR}}/{{.cluster}}/bootstrap/talos/assets/{{.hostname}}.secret.sops.yaml |
envsubst |
talosctl --context {{.cluster}} apply-config --mode={{.mode}} --nodes {{.hostname}} --file /dev/stdin sops --decrypt ~/Artemis-Cluster/kubernetes/flux/vars/cluster-secrets.secret.sops.yaml | kubectl apply --server-side --filename -

talosctl apply-config -f controlplane-thor.yaml -n 10.10.30.3 --insecure talosctl apply-config -f controlplane-baldr.yaml -n 10.10.30.4 --insecure

PFsense

Designations

MikroTik VLAN

https://www.youtube.com/watch?v=-WAKIyNCfA4

K3s Installation

FluxCD Bootstrap

Commands to bootstrap and install flux

Bootstrap Commands

brew install fluxcd/tap/flux

kubectl apply --server-side --kustomize ~/Artemis-Cluster/kubernetes/bootstrap/flux

# Install age.key secret
cd ~/.sops
cat age.agekey |
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin

# Install github and cluster secrets
sops --decrypt ~/Artemis-Cluster/kubernetes/bootstrap/flux/github.secret.sops.yaml | kubectl apply --server-side --filename -
sops --decrypt ~/Artemis-Cluster/kubernetes/flux/vars/cluster-secrets.secret.sops.yaml | kubectl apply --server-side --filename -

#Install kustomizations for the cluster
kubectl apply --server-side --filename ~/Artemis-Cluster/kubernetes/flux/vars/cluster-settings.yaml
kubectl apply --server-side --kustomize ~/Artemis-Cluster/kubernetes/flux/config

Taskfile

Boostrap from scratch and will do all the following boostrap commands:

task flux:boostrap

Install Fluxctl:

task flux:boostrap-fluxctl

Create all secrets:

task flux:bootstrap-createagesecrets

Apply all repos:

task flux:bootstrap-applyrepos:

Cilium Install

brew install cilium-cli

helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium --version 1.15.5 \
   --namespace kube-system \
   --reuse-values \
   --set operator.replicas=1 \
   --set kubeProxyReplacement=true \
   --set l2announcements.enabled=true \
   --set k8sClientRateLimit.qps=32 \
   --set k8sClientRateLimit.burst=60 \
   --set kubeProxyReplacement=strict \
   --set k8sServiceHost=10.10.99.201 \
   --set k8sServicePort=6443 \
   --set gatewayAPI.enabled=true

External Secrets

  • add repo
  • create name space - external-secrets
  • install crds from eso
  • run helm chart for bwsm-eso
  • install secrets
  • install bwsm eso to create connection to bitwarden
    • get secret manager token for service acc
  • create cluster secret store
  • only proceed once its valid and ready?

TODO

Clean this section up in the docs branch

TrueNAS

Links

Repo References