Browse Source

Added MetalLB + OPNsense post

main
Victor Roest 1 year ago
parent
commit
4aac16a5f4
Signed by: 0x76 GPG Key ID: A3923C699D1A3BDA
  1. BIN
      assets/images/opnsense-bgp-neighbour.png
  2. BIN
      assets/images/opnsense-bgp-prefix-list.png
  3. BIN
      assets/images/opnsense-routing-bgp.png
  4. BIN
      assets/images/opnsense-routing-general.png
  5. BIN
      assets/images/welcome-to-nginx.png
  6. 147
      content/posts/metallb-opnsense.md

BIN
assets/images/opnsense-bgp-neighbour.png (Stored with Git LFS)

Binary file not shown.

BIN
assets/images/opnsense-bgp-prefix-list.png (Stored with Git LFS)

Binary file not shown.

BIN
assets/images/opnsense-routing-bgp.png (Stored with Git LFS)

Binary file not shown.

BIN
assets/images/opnsense-routing-general.png (Stored with Git LFS)

Binary file not shown.

BIN
assets/images/welcome-to-nginx.png (Stored with Git LFS)

Binary file not shown.

147
content/posts/metallb-opnsense.md

@ -1,24 +1,151 @@
---
title: "MetalLB OPNsense"
date: 2021-04-28T11:40:57+02:00
draft: true
TocOpen: true
title: "Setting up OPNsense with MetalLB on Kubernetes"
summary: "A post on how to configure OPNsense FRR and MetalLB in BGP mode."
date: 2021-05-08T11:40:57+02:00
draft: false
TocOpen: false
keywords:
- bgp
- kubernetes
- metallb
- opnsense
- frr
- opnbgpd
- pfsense
- loadbalancer
- networking
- tutorial
tags:
- Kubernetes
- OPNsense
- Networking
categories:
- tutorial
- Homelab
---
Kubernetes doesn't offer an implemention for [Services of type Loadbalancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/)
out of the box for baremetal clusters [^1]. This is where [MetalLB](https://metallb.universe.tf) comes into the picture, it adds what kubernetes is missing here.
MetalLB has two modes of operating, in layer 2 mode using ARP requests or with BGP. The BGP mode is needed for 'true' load balancing [^2].
... intro ...
So this is why one might want to use MetalLB, but for proper BGP functionality you of course also need a BGP router to do the actual balancing.
For this part of the setup I am using the free opensource router software [OPNsense](https://opnsense.org/) together with its FRR (Free Range Routing) package
to provide the BGP functionality.
## OPNsense Setup
Assuming you already have OPNsense set up just add the [FRR Package](https://docs.opnsense.org/manual/dynamic_routing.html) as described in the OPNsense docs.
After this is done you can go to `Routing > General` and enable the routing deamon.
{{< img path="images/opnsense-routing-general.png" alt="Screenshot of the settings under Routing > General showing the enable box ticked" >}}
### BGP General Settings
Now navigate to the BGP general settings (`Routing > BGP`) and enable this as well. You should also choose an AS number and enter the network mask here.
I'd recommend choosing an ASN that falls into the private range as described by [RFC6996](https://tools.ietf.org/html/rfc6996),
that is a number between 64512 and 65534. You also need to set the routing distribution here, I've set it connected and kernel routes.
{{< img path="images/opnsense-routing-bgp.png" alt="Screenshot showing the OPNsense BGP General routing tab" >}}
### BGP Prefix List config
By default FRR won't permit any route announcements so we need to add a rule allowing it from the peers we'll define later.
To do so head to the "Prefix List" tab and add a rule with action permit to network any.
{{< img path="images/opnsense-bgp-prefix-list.png" alt="A screenshot showing a prefix list configuration as described" >}}
### BGP Neighbour config
Now that we have configured a prefix list we can add some neigbours that will use that rule.
Add neigbours with as `Peer-IP` the IP of that kubernetes worker node, a Remote ASN that is different from OPNsense's and the
previously made prefix list for in and output.
{{< img path="images/opnsense-bgp-neighbour.png" alt="A screenshot showing a neighbour configuration as described" >}}
Repeat this for as many kubernetes nodes you have keeping the ASN the same and changing the Peer-IP.
Also make sure all neighbours show up as enabled.
This is all we need to setup on the router's side.
## Kubernetes Setup
For the MetalLB mainly just follow the [official documentation](https://metallb.universe.tf/).
## OPNsense Setup
For the ConfigMap you'll want something like this
```yml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 10.10.10.1
peer-asn: 64512
my-asn: 64513
address-pools:
- name: default
protocol: bgp
addresses:
- 10.10.10.11-10.10.10.250
```
* `peer-address` being the IP of the router
* `peer-asn` being the ASN of the router you configured
* `my-asn` the ASN you configured for the neighbours.
* `addresses` being the range(s) MetalLB is allowed to use to allocate IPs from.
## Finishing Up
Now that everything should be configured we can test this out by making a test deployment like so:
```yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-nginx
spec:
selector:
matchLabels:
run: test-nginx
replicas: 3
template:
metadata:
labels:
run: test-nginx
spec:
containers:
- name: test-nginx
image: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: test-nginx
labels:
run: test-nginx
spec:
type: LoadBalancer
ports:
- port: 80
protocol: TCP
selector:
run: test-nginx
```
After a few moments you should be able to get the provisioned IP like so:
```sh
; kubectl describe service test-nginx | grep "LoadBalancer"
Type: LoadBalancer
LoadBalancer Ingress: 10.10.10.16
```
And when navigating to this IP you should see the familiar nginx welcome screen
{{< img path="images/welcome-to-nginx.png" alt="A screenshot of the default nginx welcome screen" >}}
Congratulations! You now have OPNsense running FRR connected to MetalLB running in Kubernetes to dynamically provision IP addreses
and loadbalance services&nbsp;&nbsp;:tada:.
---
## References
* https://metallb.universe.tf/concepts/bgp/
Some resources I've used to compile this post which may be useful to you
* https://metallb.universe.tf/
* https://blog.matrixpost.net/set-up-dynamic-routing-with-frr-free-range-routing-in-pfsense-openbgpd-now-depricated/
* https://007ba7.us/howto/metallb/
* https://www.danmanners.com/posts/2019-02-pfsense-bgp-kubernetes/
* https://devopstales.github.io/home/k8s-metallb-bgp-pfsense/
* https://docs.opnsense.org/manual/dynamic_routing.html
[^1]: These are usually only available in cloud environments like GCP, AWS, Azure, etc.
[^2]: For more information about the different modes please look at the [MetalLB docs](https://metallb.universe.tf/concepts/).

Loading…
Cancel
Save