EXPEDIA GROUP TECHNOLOGY — ENGINEERING

As businesses grow, there is an eventual need to scale the system. This proliferation could also be due to other reasons such as multi-provider strategies, geographical constraints, and computational usage. Most of these systems, at present, exploit some kind of container technology, such as Kubernetes, which help them manage and orchestrate their workloads on different worker nodes which constitute a cluster.

Need for a Multi-cluster Architecture

There are many scenarios, depending upon the requirements, where the need for multiple clusters becomes a necessity, few such scenarios are as follows —

• Low latency: The presence of Kubernetes clusters in multiple regions minimises the latency as users are served content from clusters nearest to their locations.
• Fault isolation: Multiple small clusters instead of a single large cluster simplifies fault isolation in case of failure.
• Scalability: User demand drives the scalability needs of a system.
• Hybrid cloud: Prevent provider lock-in by having multiple clusters on different cloud providers or on-premises data centres.
• Business Isolation: Maintaining separate clusters for different business domains facilitates the decoupling of services and provides better performance when compared to the multi-tenant architecture that relies solely on the presence of namespaces.

Why Federation?

Federation simplifies the management of multiple clusters and coordination of different resource configurations across clusters in a multitude of data centres. It also provides the following benefits —

• Allows us to sync specific resources across specific clusters.
• High availability as federation reduces the impact of cluster failure.
• Facilitates the auto-configuration of load balancers, network policies and DNS servers with the cluster backends.
• Prevents cluster provider lock-in since federation simplifies the cross-cluster application migration process.

In Kubernetes, federation is implemented through Kubefed. For Kubefed, two versions have been released and the earlier version has already been deprecated. More information related to Kubefed v1 could be found here.

Architecture of Kubefed v2

Before moving on to the architecture, there are some terms related to Kubefed which are important for its understanding.

1. Federate — This involves the creation of a common interface to a pool of clusters which could then be used for the deployment of K8S applications across the same clusters.
2. Kubefed Control Plane — It consists of an operator running on a master cluster, and is responsible for adding Kubefed CRDs on the master cluster which would then handle the propagation process. The control plane uses the majorly used CRD + Controller implementation, which could be installed directly onto the existing Kubernetes Clusters.
3. Kubefed API — This controls all the cluster-wide configurations and the scope of their propagation.
4. Host Cluster — This Cluster executes the Kubefed Control Plane and exposes the Kubefed API.
5. Member Cluster — This Cluster is registered to the Kubefed API in the Host cluster and the Kubefed controllers contain the authentication credentials for this cluster.
6. Propagation — This could be defined as a mechanism involving the distribution of resources across federated clusters.
7. Endpoint — This represents a DNS resource record.

The Kubefed configuration consists of two components — Type and Cluster Configuration.

Cluster Configuration

This enables us to configure what clusters are to be involved in the federation activity. The register information needed for the cluster registration by the control plane is provided through the same.

Type Configuration

This allows us to configure what API types have to be handled by Kubefed and how that resource appears in the different federated clusters through propagation. Each type configuration is a CRD and could be divulged into the following three basic components —

1. Templates — configures the basic definition of the resource to be propagated across the federated clusters.
2. Placement — specify clusters where the federation should take place.
3. Override — include cluster-specific template information that could be overridden during the propagation process.

There are some higher-level critical APIs which use the aforementioned concepts as a foundation —

1. Status —fetches the status of resources propagated across the federated clusters.
2. DNS — handles the federated services and ingresses and uses the DNS Provider for maintaining DNS entries.
3. Policy — similar to overriding, allows which clusters a resource could be propagated.
4. Scheduling — facilitates the decision-making using the status of propagated resources, it also defines how deployments and replica sets are deployed on federated clusters.

Demo — Federating a GO App

To understand the whole process of federation, we will be performing the following demo using sandbox clusters. Before starting, there are some prerequisites which need to be installed.

1. Kubectl — This provides a command-line interface for executing and managing Kubernetes clusters.
2. Helm — This is a Kubernetes package manager used for installing resources and managing workloads on Kubernetes clusters.
3. Kubefedctl— This is the command-line utility of Kubefed.
4. Kind — This provides us with a local Kubernetes environment.

In this demo, we will be installing kubefed on a host cluster, registering multiple clusters to this host and creating resources which would then be propagated to the federated clusters.

Creating a Cluster

Create two clusters with the following command while keeping the cluster names as cluster-1 and cluster-2.

$ git clone https://github.com/kubernetes-sigs/kubefed.git
$ cd kubefed
$ bash scripts/create-clusters.sh

To switch between different cluster contexts, use the following commands —

$ kubectl config current-context
$ kubectl config get-contexts
$ kubectl config use-context host-cluster

Installing Kubefed Control Plane on the host

Here, we will be using helm for installing the Kubefed control plane onto our host cluster as per the steps provided in the documentation.

• Add the Kubefed repo chart to your local environment.

$ helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts$ helm repo list

• Install the chart.

$ helm --namespace kube-federation-system upgrade -i kubefed kubefed-charts/kubefed --create-namespace

• Verify installation by checking the installed resources in the host-cluster.

Registering clusters to the Kubefed host-cluster

For registering the members to the Kubefed control plane, we need to use the following Kubefedctl command for all of the clusters. ( Fill in the appropriate cluster names in place of the placeholders. )

$ kubefedctl join { member-cluster } --cluster-context { member-cluster } --host-cluster-context { host-cluster } --v=2

The cluster registration could be checked using the following —

$ kubectl -n kube-federation-system get kubefedclusters

Propagate resources to the member clusters

Set the context to the host cluster using —

kubectl config use-context cluster-1

Create a new file manifest.yaml in the current directory and add the following configuration to the same file.

Use kubectl apply -f manifest.yaml to install the resources on all of the member clusters.

The propagation could be verified using the kubectl describe command on any of the propagated resources.

kubectl describe federateddeployment go-app --namespace go-app

Cluster Overrides and Placement

Overrides provide us with the ability to give cluster-specific variations/configurations while a resource is propagated across the federated clusters. An override consists of three components —

1. Path — A path to the field that needs to be updated in the configuration.
2. Op — The operation could be either add, remove or replace.
3. Value —The new value that has to be updated.

Cluster Placement

With the help of the placement field, we could specify which subset of the federated clusters should a resource be propagated. This includes two sub-fields —

1. .clusterSelector — Clusters are filtered on the basis of the labels provided to the field. An empty field {} implies that propagation occurs to all of the clusters.
2. .clusters — Provide a list of all clusters. An empty list [] implies no propagation activity would happen.

The following is an example of how overrides and placement could be included in the configuration of a federated CRD.

Conclusion

Through this article, we recognised the need for a multi-cluster architecture and how federation could simplify the same for our user requirements. We accomplished it through a quick demo where a go-app was deployed from the Kubefed control plane to the member clusters.