ServiceLB has some limitations that one may want to alleviate. This guide walks through replacing k3s’s default ServiceLB with MetalLB. This guide covers MetalLB’s architecture (controller and speakers), installation via Helm, and configuration of IP address pools and layer 2 advertisements, how to use it and the Traefik ingress configuration.

Rationale for using MetalLB for k3s

By default, k3s ships with service LB, a simple load balancer that leverages iptables to route external traffic to the Service’s clusterIP via a DaemonSet for each service.

However, ServiceLB has two limitations. As it cannot assign IP addressses dedicated to a LoadBalancer service, it uses the same external IP for each service, thus inducing potential port conflicts. Also, the client IP address is not preserved. The IP address seen by the application is always the ServiceLB pod that carries the traffic, rather than the “real” IP address of the user’s device, even with externalTrafficPolicy: Local. This makes it impossible to block malicous or spammy requests based on their IP address.

MetalLB

How does it work #

The goal of MetalLB is to provide load balancing for bare metal kubernetes clusters, which, as opposed to their cloud-based counterparts, often lack an implementation of a LoadBalancer to expose kubernetes traffic to the outside world. ¹

Metallb is composed of a one Controller and speakers on each kubernetes node (via a DaemonSet). The speakers are host-networked pods that are responsible for handling inbound traffic. memberlist is used to maintain a list of healthy spearkers. The controller is responsible for IP address assignation for each LoaBalancer service.

In L2 mode, once a speaker is elected as the leader for handling a Service’s traffic, it announces itself through GARP anouncements and replies to ARP requests for the IP address assigned to the service.

Installation #

Start each k3s server (not relevant for the agents) with the --disable=servicelb argument to disable the default ServiceLB LoadBalancer. This can be done with the INSTALL_K3S_EXEC="--disable=servicelb" environment variable for the k3s installation script.

Install the helm chart through the k3s HelmController. The MetalLB version is pinned to the latest one at the time of writing and disable the FRR k8s part responsible for the BGB mode of MetalLB (outside the scope of this article)

 1---
 2apiVersion: helm.cattle.io/v1
 3kind: HelmChart
 4metadata:
 5  name: metallb
 6  namespace: metallb
 7spec:
 8  chart: metallb
 9  version: v0.16.1
10  repo: https://metallb.github.io/metallb
11  valuesContent: |-
12    frrk8s:
13      enabled: false

You can either apply this manifest or put it in /var/lib/rancher/k3s/server/manifest of your k3s server so that it picks it up for you.

Configuration of MetalLB #

Configure an IPAddressPool available to MetalLB. This address poll must be outside the range of your DHCP or other dynamically allocated IP addresses to avoid any potential conflict. And associate the IPAddressPool to a layer 2 advertisement.

 1---
 2apiVersion: metallb.io/v1beta1
 3kind: IPAddressPool
 4metadata:
 5  name: pool
 6  namespace: metallb
 7spec:
 8  addresses:
 9    - 192.168.1.201-192.168.1.250
10---
11apiVersion: metallb.io/v1beta1
12kind: L2Advertisement
13metadata:
14  name: l2-advertisement
15  namespace: metallb
16spec:
17  ipAddressPools:
18    - pool

Service creation #

To use MetalLB, create a Service of the type LoadBalancer, for example:

 1apiVersion: v1
 2kind: Service
 3metadata:
 4  labels:
 5    app: myapp
 6  name: myapp-service
 7spec:
 8  ports:
 9    - name: http
10      port: 8000
11      protocol: TCP
12      targetPort: 8000
13  selector:
14    app: myapp
15  externalTrafficPolicy: Local
16  type: LoadBalancer

The MetalLB controller will assign an IP address from the IPAddressPool, and a layer 2 speaker will advertise this IP address and handle the traffic to the pods. The assigned IP address will be shown in the external IP field of the service

$  kubectl -n get service myapp-service -o jsonpath="{.items[*].status.loadBalancer.ingress[0].ip}"
192.168.1.205

```bash
$ sudo arping 192.168.1.205
ARPING 192.168.1.205 from <node ip> <interface>
Unicast reply from 192.168.1.205 [<mac>]  7.202ms

Configuration of the Traefik Ingress

To see the “real” client IP adddress on the traefik ingress, configure the externalTrafficPolicy of the LoadBalancer service created for the K3S ingress:

 1apiVersion: helm.cattle.io/v1
 2kind: HelmChartConfig
 3metadata:
 4  name: traefik
 5  namespace: kube-system
 6spec:
 7  valuesContent: |-
 8    service:
 9      spec:
10        externalTrafficPolicy: Local

The externalTrafficPolicy of the service configures how traffic flows from outside the k3S cluster to the pods. If the externalTrafficPolicy is set to Cluster (the default value), traffic is forwarded to any pod in the cluster. If set to Local, traffic is forward to the pods hosted by the node receiving the traffic. While it allows an even distribution of traffic within the cluster, the default Cluster value hides the source IP address (as it needs NAT), which is therefore not visible from the pod. Which is why it is changed here for the ingress helm chart.