6 changed files with 152 additions and 10 deletions
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@ -1,24 +1,151 @@ |
|||
--- |
|||
title: "MetalLB OPNsense" |
|||
date: 2021-04-28T11:40:57+02:00 |
|||
draft: true |
|||
TocOpen: true |
|||
title: "Setting up OPNsense with MetalLB on Kubernetes" |
|||
summary: "A post on how to configure OPNsense FRR and MetalLB in BGP mode." |
|||
date: 2021-05-08T11:40:57+02:00 |
|||
draft: false |
|||
TocOpen: false |
|||
keywords: |
|||
- bgp |
|||
- kubernetes |
|||
- metallb |
|||
- opnsense |
|||
- frr |
|||
- opnbgpd |
|||
- pfsense |
|||
- loadbalancer |
|||
- networking |
|||
- tutorial |
|||
tags: |
|||
- Kubernetes |
|||
- OPNsense |
|||
- Networking |
|||
categories: |
|||
- tutorial |
|||
- Homelab |
|||
--- |
|||
Kubernetes doesn't offer an implemention for [Services of type Loadbalancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/) |
|||
out of the box for baremetal clusters [^1]. This is where [MetalLB](https://metallb.universe.tf) comes into the picture, it adds what kubernetes is missing here. |
|||
MetalLB has two modes of operating, in layer 2 mode using ARP requests or with BGP. The BGP mode is needed for 'true' load balancing [^2]. |
|||
|
|||
... intro ... |
|||
So this is why one might want to use MetalLB, but for proper BGP functionality you of course also need a BGP router to do the actual balancing. |
|||
For this part of the setup I am using the free opensource router software [OPNsense](https://opnsense.org/) together with its FRR (Free Range Routing) package |
|||
to provide the BGP functionality. |
|||
|
|||
## OPNsense Setup |
|||
Assuming you already have OPNsense set up just add the [FRR Package](https://docs.opnsense.org/manual/dynamic_routing.html) as described in the OPNsense docs. |
|||
|
|||
After this is done you can go to `Routing > General` and enable the routing deamon. |
|||
{{< img path="images/opnsense-routing-general.png" alt="Screenshot of the settings under Routing > General showing the enable box ticked" >}} |
|||
|
|||
### BGP General Settings |
|||
Now navigate to the BGP general settings (`Routing > BGP`) and enable this as well. You should also choose an AS number and enter the network mask here. |
|||
I'd recommend choosing an ASN that falls into the private range as described by [RFC6996](https://tools.ietf.org/html/rfc6996), |
|||
that is a number between 64512 and 65534. You also need to set the routing distribution here, I've set it connected and kernel routes. |
|||
{{< img path="images/opnsense-routing-bgp.png" alt="Screenshot showing the OPNsense BGP General routing tab" >}} |
|||
|
|||
### BGP Prefix List config |
|||
By default FRR won't permit any route announcements so we need to add a rule allowing it from the peers we'll define later. |
|||
To do so head to the "Prefix List" tab and add a rule with action permit to network any. |
|||
|
|||
{{< img path="images/opnsense-bgp-prefix-list.png" alt="A screenshot showing a prefix list configuration as described" >}} |
|||
|
|||
### BGP Neighbour config |
|||
Now that we have configured a prefix list we can add some neigbours that will use that rule. |
|||
Add neigbours with as `Peer-IP` the IP of that kubernetes worker node, a Remote ASN that is different from OPNsense's and the |
|||
previously made prefix list for in and output. |
|||
|
|||
{{< img path="images/opnsense-bgp-neighbour.png" alt="A screenshot showing a neighbour configuration as described" >}} |
|||
|
|||
Repeat this for as many kubernetes nodes you have keeping the ASN the same and changing the Peer-IP. |
|||
Also make sure all neighbours show up as enabled. |
|||
|
|||
This is all we need to setup on the router's side. |
|||
|
|||
## Kubernetes Setup |
|||
For the MetalLB mainly just follow the [official documentation](https://metallb.universe.tf/). |
|||
|
|||
## OPNsense Setup |
|||
For the ConfigMap you'll want something like this |
|||
```yml |
|||
apiVersion: v1 |
|||
kind: ConfigMap |
|||
metadata: |
|||
namespace: metallb-system |
|||
name: config |
|||
data: |
|||
config: | |
|||
peers: |
|||
- peer-address: 10.10.10.1 |
|||
peer-asn: 64512 |
|||
my-asn: 64513 |
|||
address-pools: |
|||
- name: default |
|||
protocol: bgp |
|||
addresses: |
|||
- 10.10.10.11-10.10.10.250 |
|||
``` |
|||
* `peer-address` being the IP of the router |
|||
* `peer-asn` being the ASN of the router you configured |
|||
* `my-asn` the ASN you configured for the neighbours. |
|||
* `addresses` being the range(s) MetalLB is allowed to use to allocate IPs from. |
|||
|
|||
## Finishing Up |
|||
Now that everything should be configured we can test this out by making a test deployment like so: |
|||
```yml |
|||
apiVersion: apps/v1 |
|||
kind: Deployment |
|||
metadata: |
|||
name: test-nginx |
|||
spec: |
|||
selector: |
|||
matchLabels: |
|||
run: test-nginx |
|||
replicas: 3 |
|||
template: |
|||
metadata: |
|||
labels: |
|||
run: test-nginx |
|||
spec: |
|||
containers: |
|||
- name: test-nginx |
|||
image: nginx |
|||
ports: |
|||
- containerPort: 80 |
|||
--- |
|||
apiVersion: v1 |
|||
kind: Service |
|||
metadata: |
|||
name: test-nginx |
|||
labels: |
|||
run: test-nginx |
|||
spec: |
|||
type: LoadBalancer |
|||
ports: |
|||
- port: 80 |
|||
protocol: TCP |
|||
selector: |
|||
run: test-nginx |
|||
``` |
|||
After a few moments you should be able to get the provisioned IP like so: |
|||
```sh |
|||
; kubectl describe service test-nginx | grep "LoadBalancer" |
|||
Type: LoadBalancer |
|||
LoadBalancer Ingress: 10.10.10.16 |
|||
``` |
|||
|
|||
And when navigating to this IP you should see the familiar nginx welcome screen |
|||
{{< img path="images/welcome-to-nginx.png" alt="A screenshot of the default nginx welcome screen" >}} |
|||
|
|||
Congratulations! You now have OPNsense running FRR connected to MetalLB running in Kubernetes to dynamically provision IP addreses |
|||
and loadbalance services :tada:. |
|||
|
|||
--- |
|||
|
|||
## References |
|||
* https://metallb.universe.tf/concepts/bgp/ |
|||
Some resources I've used to compile this post which may be useful to you |
|||
* https://metallb.universe.tf/ |
|||
* https://blog.matrixpost.net/set-up-dynamic-routing-with-frr-free-range-routing-in-pfsense-openbgpd-now-depricated/ |
|||
* https://007ba7.us/howto/metallb/ |
|||
* https://www.danmanners.com/posts/2019-02-pfsense-bgp-kubernetes/ |
|||
* https://devopstales.github.io/home/k8s-metallb-bgp-pfsense/ |
|||
* https://docs.opnsense.org/manual/dynamic_routing.html |
|||
|
|||
|
|||
[^1]: These are usually only available in cloud environments like GCP, AWS, Azure, etc. |
|||
[^2]: For more information about the different modes please look at the [MetalLB docs](https://metallb.universe.tf/concepts/). |
|||
|
Loading…
Reference in new issue