exec format error in Kubernetes environment

Root cause of the Exec Format Error when starting container in Kubernetes cluster or ECS environment

The exec format error typically arises when trying to run a binary file that is compiled for a different architecture or operating system.

Kubernetes operates in a clustered environment where nodes can have varying architectures. If a pod's container is built for a different architecture than the node it's scheduled on, this error can occur.

For example, imagine trying to execute an ARM-based container on an AMD node. The system would encounter an exec format error because the instruction set of the ARM container doesn't match the AMD architecture.

Most common use case is that you build a Docker image on MacOS M1 with ARM architecture , and using that on in EKS cluster where nodes are AMD based. Using Buildx Docker extension, and specifying proper architecture matching architecture of your nodes - typically fixes the issue

MacOS :

  docker buildx create --name multiarch --driver docker-container --use

This command creates a new instance of the Buildx builder with the name multiarch.

--driver docker-container specifies the driver used by the builder. In this case, it's using the docker-container driver, which allows it to build images inside a containerized environment.

--use sets the newly created builder as the current active builder.

  docker buildx build --platform linux/amd64 -t image-name:v1 ./

This command initiates the build process using the Buildx builder named multiarch.

--platform linux/amd64 specifies that the image should be built for the AMD64 architecture, which is the standard x86_64 architecture.

-t image-name:v1 sets the name and tag for the resulting image. In this example, it's named image-name with the tag v1.

./ specifies the build context. It tells Docker where to find the Dockerfile and other resources needed for the build.

Re-uploading the Docker image with proper AMD architecture to ECR

Set the variables to substitude environmental variables , corresponding to your AWS account

export AWS_ACOUNT=1234567899;
export REGION=us-east-1;
export REPO=example-repo:v1;

Authenticate Docker with ECR

aws ecr get-login-password \
--region $REGION \
| docker login \
    --username AWS \
    --password-stdin $AWS_ACOUNT.dkr.ecr.$REGION.amazonaws.com

Build and push AMD Docker image to ECR

docker buildx build --push --platform linux/amd64 -t $AWS_ACOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO ./