I have a comprehension question in regards to on-prem loadbalancing / ingress. On cloud-providers you could create a service type loadbalancer which would create a LB infront of your Kubernetes cluster. On-prem with VCenter this doesn’t exist. Which raises the question: how can I do ingress with Kubermatic Kubernetes Platform on-prem? Would I deploy an ingress controller on the seed cluster and all ingress communication goes to the seed or would I place this on the user cluster? Do you have some recommendations?
Normally we use MetalLB for seed and user cluster load balacing on-prem (https://metallb.universe.tf/installation/#installation-by-manifest) Therefore you need L2 or BGP enabled, see https://metallb.universe.tf/configuration/. For automatic configuration, we create a Kubermatic Addon (https://docs.kubermatic.com/kubermatic/master/advanced/addons/), where you could enter a virtual-IP range for the MetalLB service range that gets used when a new service type LoadBalancer is provisioned.
For the ingress, you can use the normal https://github.com/kubernetes/ingress-nginx/ which uses a Service type LoadBalancer for inbound connections. You could also create an addon for this.
We will soon launch some community addons here https://github.com/kubermatic-labs/community-components where you will find these components.
For the API server in front of the seed cluster, a simple go between setup will do the trick, see https://github.com/kubermatic/kubeone/tree/master/examples/terraform/vsphere