Table of Contents

Fedora Homelab - This article is part of a series.

Part 1: Setting Up a Secure Fedora Homelab with Teleport & Cloudflare

Part 2: This Article

Introduction
#

Now that our homelab is setup, we are going to be running a kubernetes cluster. For our purposes we have a couple of options but we will be going with Kind. If you are curious these are the various ways you can run a kubernetes cluster locally:

Minikube: While excellent for beginners, Minikube primarily focuses on running a single-node cluster inside a virtual machine, which is less ideal for simulating the realistic multi-node scenarios (like HA, network policies, and advanced scheduling) that our project will require.
k3d: k3d is fantastic for running k3s (a lightweight K8s) and is extremely fast, but kind provides a cluster using the full, upstream Kubernetes binaries, offering a more conformant and “standard” environment for testing features exactly as they’d behave in production.
kind: We chose kind (Kubernetes in Docker) because it excels at creating lightweight, multi-node clusters locally using Docker containers, perfectly matching our need to simulate a real cluster (e.g., 1 control plane + 2 workers) for learning advanced networking, scaling, and chaos engineering tasks.
MicroK8s: While MicroK8s does seem like a very good option for setting up long term clusters with it’s ease of use. It would make our life a bit too easy, and without a bit of pain we wouldn’t learn much so this option is out of the window.

Creating Kind Cluster
#

Kind uses docker to create a container for each of the nodes, so we will first need to install docker:

sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable docker

Now for creating the cluster we will first need to make a kind config file, we will have 1 control plane and 2 worker nodes for simulating a realistic multi-node cluster:

# kind-config.yaml - A cluster with 1 control-plane node and 2 worker nodes
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  ipFamily: dual
nodes:
- role: control-plane
- role: worker
- role: worker

disableDefaultCNI: This will disable the default CNI (kindnet) which is very simple as we will be using cilium which is a production grade advanced CNI.
ipFamily: dual enables iPv6 which we will be learning as well. Now to create the cluster:

kind create cluster --name k8s-lab --config kind-config.yaml

Now you can additionally add this cluster to teleport using this guide.

Installing and configuring cilium
#

Cilium is a complete networking, observability, and security platform that uses eBPF instead of iptables, which is used by the default kindnet. This makes it highly performant and reduces latency. It also provides observability through Hubble.

eBPF is a revolutionary technology inside the Linux kernel that lets you run sandboxed, custom programs directly within the kernel itself without changing the kernel’s source code.

Cilium has these components:

Cilium Agent (cilium): This is a daemonset that runs on every node. It’s the core of Cilium, managing the eBPF programs, enforcing network policies, and handling traffic.
Operator (cilium-operator): This is a central deployment that handles tasks for the entire cluster, like IP address management (IPAM) and coordinating tasks that don’t need to run on every single node.
Hubble Components (hubble-relay, hubble-ui): These are dedicated pods that collect the observability data from the agents (relay) and present it in a graphical interface (UI

Installing cilium using helm
#

For now, we’ll install Cilium using Helm and later manage it through Argo CD. If Argo CD were running in a separate management cluster, it could deploy Cilium directly. However, since Argo CD runs within this cluster, we first need a CNI installed otherwise, Argo CD’s pods won’t receive IP addresses and cannot start without networking.

Install cilium cli:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Make values.yaml:

cni:
  exclusive: false
cluster:
  name: kind-k8s-lab
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
    service:
      type: NodePort
  # metrics:
  #   serviceMonitor:
  #     enabled: true
  #   enableOpenMetrics: true
  #   enabled:
  #     - dns
  #     - drop
  #     - tcp
  #     - flow
  #     - port-distribution
  #     - icmp
  #     - "httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction"

ipam:
  mode: kubernetes
operator:
  replicas: 1
  # prometheus:
  #   enabled: true
  #   serviceMonitor:
  #     enabled: true
routingMode: tunnel
tunnelProtocol: vxlan
ipv6:
  enabled: true
kubeProxyReplacement: "true"
k8sServiceHost: "172.18.0.3"
k8sServicePort: 6443

# prometheus:
#   enabled: true
#   serviceMonitor:
#     enabled: true

You can get IP for k8sServiceHost using the command docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' kind-lab-control-plane (replace with the name of your control plane )

We will uncomment the lines with metrics after we install our observability stack.

Install cilium CNI to our cluster with:

helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium \
  -n kube-system \
  -f values.yaml \
  --set-string "argo-cd.argoproj.io/managed-by=cilium"

This automatically detected that we are using kind and installed successfully.

Now check if cilium was properly installed:

-> cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet              cilium-envoy             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 3
                       cilium-envoy             Running: 3
                       cilium-operator          Running: 1
                       clustermesh-apiserver
                       hubble-relay
Cluster Pods:          4/4 managed by Cilium
Helm chart version:    1.18.1
Image versions         cilium             quay.io/cilium/cilium:v1.18.1@sha256:65ab17c052d8758b2ad157ce766285e04173722df59bdee1ea6d5fda7149f0e9: 3
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.34.4-1754895458-68cffdfa568b6b226d70a7ef81fc65dda3b890bf@sha256:247e908700012f7ef56f75908f8c965215c26a27762f296068645eb55450bda2: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.18.1@sha256:97f4553afa443465bdfbc1cc4927c93f16ac5d78e4dd2706736e7395382201bc: 1

To verify if our cluster has proper network connectivity:

➜  cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [kind-k8s-lab] Creating namespace cilium-test-1 for connectivity check...
✨ [kind-k8s-lab] Deploying echo-same-node service...
✨ [kind-k8s-lab] Deploying DNS test server configmap...
✨ [kind-k8s-lab] Deploying same-node deployment...
✨ [kind-k8s-lab] Deploying client deployment...
✨ [kind-k8s-lab] Deploying client2 deployment...
✨ [kind-k8s-lab] Deploying client3 deployment...
✨ [kind-k8s-lab] Deploying echo-other-node service...
✨ [kind-k8s-lab] Deploying other-node deployment...
✨ [host-netns] Deploying kind-k8s-lab daemonset...
✨ [host-netns-non-cilium] Deploying kind-k8s-lab daemonset...
ℹ️  Skipping tests that require a node Without Cilium
⌛ [kind-k8s-lab] Waiting for deployment cilium-test-1/client to become ready...
⌛ [kind-k8s-lab] Waiting for deployment cilium-test-1/client2 to become ready...
⌛ [kind-k8s-lab] Waiting for deployment cilium-test-1/echo-same-node to become ready...
timeout reached waiting for deployment cilium-test-1/echo-same-node to become ready (last error: only 0 of 1 replicas are available)

We can see that few pods are failing to start:

➜ kubectl get pods -n cilium-test-1
NAME                               READY   STATUS             RESTARTS      AGE
client-7b7776c86b-sfswq            1/1     Running            0             4m43s
client2-57cf4468f-mbvzp            1/1     Running            0             4m43s
client3-67f959dd9b-fcz5s           1/1     Running            0             4m43s
echo-other-node-85b6b57f54-9kngq   1/2     CrashLoopBackOff   5 (42s ago)   4m42s
echo-same-node-7fc487dc56-wc7m6    1/2     CrashLoopBackOff   5 (51s ago)   4m43s

Checking the pod logs:

Error: EMFILE: too many open files, watch '/'
    at FSWatcher.<computed> (node:internal/fs/watchers:247:19)
    at Object.watch (node:fs:2469:36)
    at /usr/local/lib/node_modules/json-server/lib/cli/run.js:163:10 {
  errno: -24,
  syscall: 'watch',
  code: 'EMFILE',
  path: '/',
  filename: '/'

The docs mention how to solve this error https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files:

Append these lines at /etc/sysctl.conf:

sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512

Run the connectivity test again and everything should be good to go

✅ [cilium-test-1] All 77 tests (684 actions) successful, 46 tests skipped, 1 scenarios skipped.

Testing dual stack
#

Since we enabled iPv6 in both our kind cluster config and cilium values.yaml. Let’s test it out and see how it works.

Deploy the nginx service. Add ipFamilyPolicy: RequireDualStack in the service specification.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ipFamilyPolicy: RequireDualStack
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Check the service to confirm it has been assigned both IPv4 and IPv6 ClusterIPs

kubectl get svc nginx-service -oyaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx-service","namespace":"default"},"spec":{"ipFamilyPolicy":"RequireDualStack","ports":[{"port":80,"protocol":"TCP","targetPort":80}],"selector":{"app":"nginx"}}}
  creationTimestamp: "2025-09-04T11:08:52Z"
  name: nginx-service
  namespace: default
  resourceVersion: "148073"
  uid: dfdb0d4e-4803-48d1-811b-a409d388f677
spec:
  clusterIP: 10.96.9.147
  clusterIPs:
  - 10.96.9.147
  - fd00:10:96::aeaa
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: RequireDualStack
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

As seen above, both IPv4 and IPv6 are assigned to the nginx service.

Now run a busybox pod.

kubectl run busybox --rm -ti --image=busybox -- sh

Connect to IPv4 address of the service

# connect to IPv4 address of the service
wget -q -O - 10.96.9.147

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Successfully connected.

Now connect to IPv6 address of the service

wget -q -O - '[fd00:10:96::aeaa]'

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Successfully connected using IPv6 too.

Conclusion
#

We’ve successfully laid the most critical piece of our homelab on top of the secure Fedora foundation from Part 1: a powerful, multi-node Kubernetes cluster.

By choosing Kind, we’ve built a setup that realistically simulates a production environment, and by installing Cilium, we’ve supercharged it with a modern, eBPF-based CNI. We didn’t just get networking, enabled dual-stack (IPv4/IPv6) support, and laid the groundwork for the advanced observability and security we’ll use later.

In the next part of this series, we’ll take the first step towards a true GitOps workflow using Argo CD. This will allow us to manage our cluster and applications declaratively, all from a git repository.

Fedora Homelab - This article is part of a series.

Part 1: Setting Up a Secure Fedora Homelab with Teleport & Cloudflare

Part 2: This Article

Secure Homelab, Part 2: Building a Multi-Node K8s Cluster with Kind and Cilium

Introduction
#

Creating Kind Cluster
#

Installing and configuring cilium
#

Installing cilium using helm
#

Testing dual stack
#

Conclusion
#

Related

Introduction #

Creating Kind Cluster #

Installing and configuring cilium #

Installing cilium using helm #

Testing dual stack #

Conclusion #

Related

Introduction
#

Creating Kind Cluster
#

Installing and configuring cilium
#

Installing cilium using helm
#

Testing dual stack
#

Conclusion
#