ECS Anywhere with AWS Copilot CLI Demo

Goal

The goal of the article is to have a locally running VM, which is enrolled into an Amazon ECS cluster using ECS Anywhere. On the local VM two containers will be running: a dummy web service listening on a port and a load balancer, which discovers the web service and exposes it based on some predefined rules.

For deployment we will use the AWS Copilot CLI. AWS Copilot doesn't support ECS Anywhere as of writing this, however, its flexible extensibility allows us to flip some CloudFormation properties and to have our containers running on 'external container instances'.

The load balancer cannot be an Application Load Balancer in this case, as the goal is to have the load balancer inside the VM. A great cloud-native alternative is Traefik, which thanks to its combination of static and dynamic configuration is a perfect fit for the task.

Benefits of this architecture

Thanks to the building blocks the following features are available for the local cluster:

1. It runs on-prem (which can be a requirement due to external constraints)
2. The container orchestration is fully cloud-native (you can use all features of ECS and SSM)
3. Rolling updates are available for the backend services due to Traffik's dynamic discovery capabilities
4. Circuit-breaker is enabled and performs automated rollbacks, if the services fail to stabilize during deployment
5. You can make use of the expressive Copilot manifests to configure your application

Prerequisites

• Install Vagrant
• Install VirtualBox
• Install AWS Copilot CLI
• Install and configure the AWS CLI

Step-by-step guide

1. Initialize Copilot app

   copilot init

2. Initialize & deploy Copilot test environment

   copilot env init -n test
   copilot env deploy -n test

3. Start Vagrant machine

   vagrant up

4. Get registration command for the local VM as an external machine:

   1. Navigate to the ECS cluster in the ECS Service
   2. Choose the right region
   3. Click the Infrastructure tab, then the Register external instances button
   4. In the popup set the Number of instances to 1, and for Instance role select Create new.
   5. Copy the registration command for Linux in the last step

5. Register the local VM as an external machine:

   vagrant ssh
   sudo -s
   # And then execute command copied from step 4
   # curl ... 

6. Make sure that the registration completes and that the instance shows up in the Infrastructure tab of the cluster.

7. Initialize & deploy the workloads

   copilot svc init -e test -n whoami
   copilot svc deploy -e test -n whoami
   copilot svc init -e test -n traefik
   copilot svc deploy -e test -n traefik

8. Now you can navigate to the Traefik dashboard to check if the Docker provider is enabled and the services are discovered:

   http://localhost:8080/
   

9. You can get a response from the whoami service by making the following request:

   curl -H 'Host: whoami.domain.com' http://localhost:8081/whoami

   Note: the port forwarding for the host port 80 with VirtualBox didn't work for me, so I redirected the VM's port 80 to port 8081 on the host.

Clean-up

1. De-register the external instance

   1. Navigate to the ECS cluster in the ECS Service
   2. Choose the right region
   3. Click the Infrastructure tab
   4. Under Container Instances select your VM, then Actions and Drain
   5. Wait until the instance disappears (this can take a while)

2. Delete the Copilot application:

   copilot app delete

3. Delete the managed node from SSM

   1. Open SSM Fleet Manager
   2. Select your VM
   3. Press Node actions, then Node settings, then Deregister this managed node

4. Delete the IAM role ecsExternalInstanceRole

5. Destroy the local machine

   vagrant destroy

References

• The article Implementing application load balancing of Amazon ECS Anywhere workloads using Traefik Proxy demonstrates a similar use case, however it suggests using the ECS discovery with Traefik, which enables putting the Traefik load balancer on one external machine and placing the backend service on a different host. It's not exactly what I was looking for, additionally, it has a major drawback: it requires the exposure of the port of the backend containers on the host, which can be a security risk, additionally it prevents rolling updates and horizontal scaling, as the port can only be bound to one container at a time.
• AWS Copilot CLI docs
• Traefik docs
  • Traefik Docker dynamic configuration reference
  • Traefik static CLI configuration