ecsdep 0.1.0a5

AWS ECS Deployment Tool With Terraform

Introduce

ECS deploy using docker compose and terraform.

You need to manage just yml file for docker compose and:

ecsdep cluster create
ecsdep service up

That's all.

Currently, ecsdep supports EC2 ECS, not Fargate.

Running Docker For Deployment

Locally

Docker contains terrform, awscli and ecsdep.

docker run -d --privileged \
    --name docker \
    -v path/to/myproject:/app \
    hansroh/dep:dind
docekr exec -it docker bash

.gitlab-ci.yml for Gitlab CI/CD

Add these lines:

image: hansroh/dep:latest
services:
  - name: docker:dind
    alias: dind-service

Prerequisitions

• AWS credebtial for ECS deployment
  • See example policy JSON
• AWS certification for ypur service domain
• AWS secret arn for private docker registry login
• AWS s3 bucket for terraform state data at your region

Make Docker Compose File For ECS Deployment

Create /app/de[/compose.ecs.yml.

This example launch 2 container - app and nginx as reverse proxy.

version: '3.9'

services:
  skitai-app:
    image: registry.gitlab.com/skitai/ecsdep
    x-ecs-pull_credentials: arn:aws:secretsmanager:ap-northeast-2:000000000:secret:gitlab/registry/mysecret-PrENMF
    build:
      context: ..
      dockerfile: dep/Dockerfile
      target: image-${SERVICE_STAGE}
    container_name: skitai-app
    logging:
      x-ecs-driver: awslogs
    x-ecs-essential: true
    deploy:
      resources:
        reservations:
          memory: "160M"
          cpus: "1024"
        limits:
          memory: "512M"
    ports:
      - 5000
    healthcheck:
      test:
        - "CMD-SHELL"
        - "wget -O/dev/null -q http://localhost:5000 || exit 1"
      interval: 30s
      retries: 3

  skitai-nginx:
    image: registry.gitlab.com/skitai/ecsdep/nginx
    x-ecs-pull_credentials: arn:aws:secretsmanager:ap-northeast-2:000000000:secret:gitlab/registry/mysecret-PrENMF
    build:
      context: ..
      dockerfile: dep/Dockerfile.nginx
    container_name: skitai-nginx
    build:
      context: ..
      dockerfile: dep/Dockerfile.nginx
    logging:
      x-ecs-driver: awslogs
    deploy:
    depends_on:
      - skitai-app
    x-ecs-wait-conditions:
      - HEALTHY
    ports:
      - 80:80
    deploy:
      resources:
        reservations:
          memory: "16M"

networks:
  ecsdep:

secrets:
  REGISTRY_USER:
    name: "arn:aws:secretsmanager:ap-northeast-2:000000000:secret:gitlab/registry/mysecret-PrENMF:username::"
    external: true


# ECS config --------------------------------------------
x-ecs-service-config:
  name: ecsdep
  stages:
    default:
      env-service-stage: "qa"
      hosts: ["qa.myservice.com"]
      listener-priority: 100

    production:
      env-service-stage: "production"
      hosts: ["myservice.com"]
      listener-priority: 101

  loadbalancing-pathes:
    - /*

  autoscaling:
    desired_count: 1
    min: 1
    max: 4
    cpu: 75
    memory: 80

  target-group:
    port: 80
    protocol: http
    health-check:
      path: "/"
      matcher: "200,301,302,404"

x-terraform:
  provider: aws
  region: ap-northeast-2
  state-backend:
    region: "ap-northeast-2"
    bucket: "states-data"
    key-prefix: "terraform/ecs-cluster"

x-ecs-cluster:
  name: mycluster
  public-key_file: "~/.ssh/id_rsa.pub"
  instance-type: t3.medium
  ami: amzn2-ami-ecs-hvm-*-x86_64-*
  autoscaling:
    min: 1
    max: 20
    desired: 1
    cpu: 80
    memory: 80
  loadbalancer:
    cert-name: myservice.com
  availability-zones: 2
  s3-cors_hosts:
    - http://localhost:5000
    - https://myservice.com
    - https://qa.myservice.com

If you want to use GPU,

services:
  skitai-app:
    ...
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

x-ecs-cluster:
  ...
  instance-type: g4dn.2xlarge

ecsdep ignores device driver, just care about count value. Make sure your cluster's instance type has GPU capability.

Testing Docker Containers

cd dep
docker-compose -f compose.ecs.yml build
docker-compose -f compose.ecs.yml up -d
docker-compose -f compose.ecs.yml down
docker-compose -f compose.ecs.yml push

Deployment

Creating/Update ECS Cluster

ecsdep -f compose.ecs.yml cluster plan
# ecsdep find compose.ecs.yml default,
ecsdep cluster plan
# if no error,
ecsdep cluster create

As a results, AWS resources will be created.

• VPC
• Application Load Balancer
• ECS Cluster
• Launch Configureation
• Security Group
• Auto Scaling Group For Cluster
• Public Accessable S3 Bucket

Deploying Service

export CI_COMMIT_SHA=latest
export SERVICE_STAGE=qa

ecsdep service plan
ecsdep service up

Whenever commanding ecsdep service up, your containers will be deployed to ECS by rolling update way.

As a results, AWS resources will be created.

• Task Definition
• Update Service and Run

Removing Service

ecsdep service down

Destroying ECS Cluster

ecsdep cluster destroy

Testable Example Project

git clone https://gitlab.com/skitai/ecsdep.git
cd ecsdep/dep
docker run -d --privileged --name dep \
    --workdir /app \
    -v ${PWD}/ecsdep:/app \
    hansroh/dep:dind
docekr exec -it dep bash

Within container,

pip3 install -U ecsdep
docker login -u <gitlab username> -p <gitlab token> registry.gitlab.com
aws configure set aws_access_key_id <AWS_ECS_ACCESS_KEY_ID>
aws configure set aws_secret_access_key <AWS_ECS_SECRET_ACCESS_KEY>

AWS access key should have proper permissions for ECS control (see above prerequisition section).

Then modify dep/compose.ecs.yml. Along this process, you should fulfill all prerequisitions.

Finally,

cd dep
./test_ecs_docker_build.sh
./test_ecsdep_deploy.sh