Creating EKS Cluster using Terraform

In today's fast-paced DevOps landscape, automation is paramount. One of the best practices for DevOps teams is to automate the provisioning and management of infrastructure. When it comes to orchestrating Kubernetes clusters on AWS, Terraform emerges as the go-to tool. In this article, we'll delve into the process of creating a Kubernetes Cluster using Terraform.

Link to Github code repo here

Prerequisites:

1. AWS Account: You must have an active AWS account. If you don't have one, you can sign up for an AWS account on the AWS website. You can create it here
2. IAM User or Role: Create an IAM (Identity and Access Management) user or role in your AWS account with the necessary permissions to create and manage EKS clusters. At a minimum, the user or role should have permissions to create EKS clusters, EC2 instances, VPCs, and related resources.
3. AWS CLI: Install and configure the AWS Command Line Interface (CLI) on your local machine. You'll use the AWS CLI to interact with your AWS account and configure your AWS credentials. You can download it here
4. Terraform Installed: Install Terraform on your local machine. You can download Terraform from the official Terraform website and follow the installation instructions for your operating system here

To create Eks Cluster we would need to create following resources:

Networking Layer

• VPC - Virtual private cloud, this is our network where our infrastructure will be located
• Availability Zones - best practice to use 2 or better 3 AZ to maintain high availability
• Private and Public Subnets for each AZ used (Eks in private subnets, load balancer proxing traffic from public to private subnets)

Compute Layer

• Elastic Kubernetes Service Cluster
• Node Groups

Let's start coding

We will use a common approach for creating resources in Terraform - modules. Terraform modules offer a modular and reusable way to define and manage infrastructure resources. This approach is considered a best practice in Terraform. We highly recommend utilizing Anton Babenko's Terraform modules. Anton Babenko is a well-known figure in the Terraform community and has contributed extensively to the ecosystem. His modules are renowned for their quality, documentation, and adherence to best practices. By leveraging his modules, we can expedite the EKS cluster creation process while maintaining a high standard of infrastructure as code.

Provider.tf

It's a good practice to start terraform code with provider.tf file to keep your code clear and organized also it allows you to change providers or their configurations more easily if you would need to change something

provider "aws" {
  region  = var.aws_region
  profile = var.aws_profile
}

As you can see we are using variables to declare AWS region and profile. Let's create Variables.tf file

variable "aws_profile" {
  description = "Set this variable if you use another profile besides the default awscli profile called 'default'."
  type        = string
  default     = "default"
}

variable "aws_region" {
  description = "Set this variable if you use another aws region."
  type        = string
  default     = "us-east-1"
}

Please note - you would need configured AWS Cli profile

Variables.tf is also a good practice so you can change valuable inputs easily in one file

VPC Module

Let's create main.tf file with the following:

locals {
  cluster_name = "${var.env}-eks-${random_string.suffix.result}"
}

resource "random_string" "suffix" {
  length  = 8
  special = false
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.2"

  name = var.vpc_name

  cidr = var.cidr
  azs  = var.aws_availability_zones

  private_subnets = var.private_subnets
  public_subnets  = var.public_subnets

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = 1
  }
}

Resource random_string.suffix creates randonm string with lenght 8 symbols which afterwards is used in our local variable to create name of our Eks Cluster.

Next we are using module called "vpc". You can find full documentation here This module has no required variables but we would declare some to override their default values. I created few more variables so it would be easier to update if needed. So we would need to add following to our variables.tf file:

variable "vpc_name" {
  description = "Vpc name that would be created for your cluster"
  type        = string
  default     = "EKS_vpc"
}

variable "aws_availability_zones" {
  description = "AWS availability zones"
  default     = ["us-east-1a", "us-east-1b", "us-east-1c"]
}

variable "cidr" {
  description = "Cird block for your VPC"
  type        = string
  default     = "10.0.0.0/16"
}

variable "env" {
  description = "it would be a prefix for you cluster name created, typically specified as dev or test"
  type        = string
  default     = "dev"
}

variable "private_subnets" {
  description = "private subnets to create, need to have 1 for each AZ"
  default     = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

variable "public_subnets" {
  description = "public subnets to create, need to have 1 for each AZ"
  default     = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
}

EKS Module

Now let's get to eks module

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.16.0"

  cluster_name    = local.cluster_name
  cluster_version = var.cluster_version

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = true
}

• cluster_name: The name of the Amazon EKS cluster, which is obtained from a local variable called local.cluster_name.
• cluster_version: The desired version of the EKS cluster, sourced from a variable named cluster_version
• vpc_id: The ID of the Virtual Private Cloud (VPC) in which the EKS cluster will be deployed. This value is obtained from the "vpc" module above by referencing module.vpc.vpc_id
• subnet_ids: A list of subnet IDs where the EKS worker nodes will be launched. These subnets are derived from the "vpc" module by accessing module.vpc.private_subnets
• cluster_endpoint_public_access: This parameter is set to "true," indicating that the EKS cluster's API server will have public access from the internet.

Eks Managed Node Group

EKS Managed Node Groups are a simplified and managed way to provision and maintain worker nodes in an Amazon EKS cluster. AWS handles the underlying infrastructure and automates tasks such as node scaling, patching, and AMI management. You only need to specify the desired instance type and desired number of nodes, and EKS takes care of the rest. Managed Node Groups are ideal for reducing operational overhead and ensuring that your nodes are always up-to-date and secure.

  eks_managed_node_groups = {
    on_demand_1 = {
      min_size     = 1
      max_size     = 3
      desired_size = 1

      instance_types = ["t3.small"]
      capacity_type = "ON_DEMAND"
    }
    spot_1 = {
      min_size     = 1
      max_size     = 3
      desired_size = 1

      instance_types = ["t3.small"]
      capacity_type  = "SPOT"
    }
  }

In this code snippet, configurations related to Amazon Elastic Kubernetes Service (EKS) managed node groups are defined using Terraform map variables. These configurations specify the settings for the managed node groups that will be attached to the EKS cluster:

1. eks_managed_node_group_defaults Configuration:

• ami_type: This is set to the value of the var.ami_type variable. It determines the Amazon Machine Image (AMI) type that will be used for the EKS managed node groups.

2. eks_managed_node_groups Configuration:

• on_demand_1 and spot_1 are two distinct EKS managed node groups that are defined within the eks_managed_node_groups map.
• For both on_demand_1 and spot_1 managed node groups:
  • min_size: Specifies the minimum number of nodes that will be maintained in the node group.
  • max_size: Specifies the maximum number of nodes that can be scaled up to in the node group.
  • desired_size: Sets the desired number of nodes to be maintained in the node group.
  • instance_types: Specifies the instance types that will be used for the worker nodes in the node group. In this case, both node groups use "t3.small" instances.
• capacity_type: Indicates the capacity type for the node group.

Let's dive deeper what does On Demand and Spot capasity type means

1. On Demand instances are always available and provide stable and predictable performance. They are suitable for workloads that require consistent performance and are not tolerant of interruptions. They are recommended for mission-critical applications, production workloads, and services where reliability and performance are paramount.

2. Spot Instances can be significantly cheaper than On-Demand instances. They are available at a discount because they use spare capacity in the AWS cloud. This makes them ideal for cost-sensitive workloads. On the other hand Spot Instances are interruptible, meaning AWS can terminate them with very little notice if the capacity is needed for On-Demand or Reserved Instances. Therefore, they are best suited for fault-tolerant workloads that can handle interruptions. Also they are often used for batch processing, data analysis, and workloads that can be distributed across multiple instances.

As example we would create both of them.

Self Managed Node Group

Self-Managed Node Groups provide full control and customization over the worker nodes in your EKS cluster. With self-managed node groups, you are responsible for provisioning, scaling, and maintaining the nodes. This option is suitable for advanced users and organizations that require fine-grained control over node configurations, custom Amazon Machine Images (AMIs), and node lifecycle management. Self-managed node groups offer flexibility but come with additional operational responsibilities compared to managed node groups.

  self_managed_node_groups = {
    one = {
      name         = "mixed-1"
      max_size     = 4
      desired_size = 2

      use_mixed_instances_policy = true
      mixed_instances_policy = {
        instances_distribution = {
          on_demand_base_capacity                  = 0
          on_demand_percentage_above_base_capacity = 10
          spot_allocation_strategy                 = "capacity-optimized"
        }

        override = [
          {
            instance_type     = "t3.small"
            weighted_capacity = "1"
          },
          {
            instance_type     = "t3.medium"
            weighted_capacity = "2"
          },
        ]
      }
    }
  }

Here we would create a self-managed node group named "mixed-1". This node group is configured with specific settings for managing worker nodes in an Amazon Elastic Kubernetes Service (EKS) cluster:

• name: Sets the name of the self-managed node group to "mixed-1."
• max_size: Specifies the maximum number of worker nodes that can be scaled up to in the node group, which is set to 4.
• desired_size: Sets the desired number of worker nodes to be maintained in the node group, which is set to 2.
• use_mixed_instances_policy: This parameter is set to "true," indicating that the node group will use a mixed instances policy, allowing it to combine different instance types for better cost optimization and performance.
• mixed_instances_policy: This block defines the mixed instances policy for the node group, which allows you to specify different instance types and their weights
• instances_distribution: Specifies the distribution of instance types within the node group. In this case:
• on_demand_base_capacity: Specifies the minimum number of On-Demand instances required, set to 0.
• on_demand_percentage_above_base_capacity: Sets the percentage of additional On-Demand instances above the base capacity, set to 10%.
• spot_allocation_strategy: Determines the strategy for allocating Spot Instances, which is set to "capacity-optimized."
• override: This block allows you to override the instance types and their weighted capacities. In this example, two instance types are specified with their respective weights:
  • "t3.small" with a weight of 1.
  • "t3.medium" with a weight of 2.

AWS Eks Addon

If you would need to use EBS volumes in your cluster, starting from cluster version 1.23 you would need to install AWS Eks EBS CSI Driver Addon in your cluster.

I will create a separate file ebs_csi.tf in a project folder for that:

data "aws_iam_policy" "ebs_csi_policy" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

module "irsa-ebs-csi" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version = "4.7.0"

  create_role                   = true
  role_name                     = "AmazonEKSTFEBSCSIRole-${module.eks.cluster_name}"
  provider_url                  = module.eks.oidc_provider
  role_policy_arns              = [data.aws_iam_policy.ebs_csi_policy.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:kube-system:ebs-csi-controller-sa"]
}

This Terraform code block fetches an AWS Managed IAM policy related to EBS CSI, and then it uses a Terraform module to create a new IAM role with IRSA enabled. This IAM role allows Kubernetes service accounts to assume it, providing them with the necessary AWS permissions to interact with EBS volumes using the EBS CSI driver. The IAM role is associated with specific Kubernetes service accounts defined by oidc_fully_qualified_subjects

1. data "aws_iam_policy" "ebs_csi_policy":
   • This block defines a data source to fetch the AWS IAM policy named "AmazonEBSCSIDriverPolicy" using its ARN (Amazon Resource Name). This policy is typically used for IAM roles associated with EBS CSI (Container Storage Interface) drivers in Kubernetes clusters.
2. module "irsa-ebs-csi":
   • This block uses a Terraform module to create an IAM role with IRSA enabled for interaction between an AWS service and an OIDC-authenticated Kubernetes service account.
   • source: Specifies the source location of the Terraform module for creating an IAM role with OIDC support
   • version: Sets the version of the Terraform module to use.
   • create_role: This parameter is set to "true," indicating that the module should create a new IAM role.
   • role_name: Specifies the name of the IAM role to be created. It includes "AmazonEKSTFEBSCSIRole-" as a prefix, followed by the EKS cluster name obtained from module.eks.cluster_name.
   • provider_url: Specifies the OIDC provider URL, which is obtained from the EKS cluster module using module.eks.oidc_provider. This URL is necessary for OIDC authentication.
   • role_policy_arns: An array containing the ARN of the AWS IAM policy. In this case, it includes the ARN fetched from the data "aws_iam_policy.ebs_csi_policy" data source.
   • oidc_fully_qualified_subjects: This specifies a list of fully qualified subjects for which the IAM role should be assumed. It typically includes Kubernetes service accounts that will use this IAM role.

Now we can create AWS EBS CSI Driver and associate it with our EKS cluster:

resource "aws_eks_addon" "ebs-csi" {
  cluster_name             = module.eks.cluster_name
  addon_name               = "aws-ebs-csi-driver"
  addon_version            = "v1.23.0-eksbuild.1"
  service_account_role_arn = module.irsa-ebs-csi.iam_role_arn
  tags = {
    "eks_addon" = "ebs-csi"
    "terraform" = "true"
  }
}

• cluster_name: Specifies the name of the EKS cluster to which this addon will be added. It uses the EKS cluster name obtained from module.eks.cluster_name
• addon_name: Sets the name of the addon to "aws-ebs-csi-driver." This addon provides support for Amazon Elastic Block Store (EBS) volumes in the EKS cluster.
• addon_version: Specifies the version of the addon to be installed. In this case, it is set to "v1.23.0-eksbuild.1"

If You want to check available versions use following command:

aws eks describe-addon-versions --addon-name aws-ebs-csi-driver --kubernetes-version 1.27 \
--query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" --output text

• service_account_role_arn: Associates an IAM role (typically the one created in the previous code block using module.irsa-ebs-csi.iam_role_arn) with the addon. This IAM role allows the addon to interact with AWS resources on behalf of the EKS cluster.
• tags: Allows you to attach metadata tags to the addon. In this example, two tags are added: "eks_addon" with a value of "ebs-csi" and "terraform" with a value of "true."

Final Step - Deployment

In order to initialize terraform and download modules run:

`terraform init` 

You can also check which resources terraform is planning to create by running:

terraform plan

To provision resources run:

terraform apply

You can find source code in our Github repo