ecs
AWS ECS container orchestration for running Docker containers. Use when deploying containerized applications, configuring task definitions, setting up services, managing clusters, or troubleshooting container issues.
Install
npx skills add https://github.com/itsmostafa/aws-agent-skills/tree/main/skills/ecs
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install itsmostafa-aws-agent-skills@llmmart
git clone https://github.com/itsmostafa/aws-agent-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole itsmostafa/aws-agent-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
AWS ECS
Amazon Elastic Container Service (ECS) is a fully managed container orchestration service. Run containers on AWS Fargate (serverless) or EC2 instances.
Table of Contents
Core Concepts
Cluster
Logical grouping of tasks or services. Can contain Fargate tasks, EC2 instances, or both.
Task Definition
Blueprint for your application. Defines containers, resources, networking, and IAM roles.
Task
Running instance of a task definition. Can run standalone or as part of a service.
Service
Maintains desired count of tasks. Handles deployments, load balancing, and auto scaling.
Launch Types
| Type | Description | Use Case |
|---|---|---|
| Fargate | Serverless, pay per task | Most workloads |
| EC2 | Self-managed instances | GPU, Windows, specific requirements |
Common Patterns
Create a Fargate Cluster
AWS CLI:
# Create cluster
aws ecs create-cluster --cluster-name my-cluster
# With capacity providers
aws ecs create-cluster \
--cluster-name my-cluster \
--capacity-providers FARGATE FARGATE_SPOT \
--default-capacity-provider-strategy \
capacityProvider=FARGATE,weight=1 \
capacityProvider=FARGATE_SPOT,weight=1
Register Task Definition
cat > task-definition.json << 'EOF'
{
"family": "web-app",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "256",
"memory": "512",
"executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::123456789012:role/ecsTaskRole",
"containerDefinitions": [
{
"name": "web",
"image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest",
"portMappings": [
{
"containerPort": 8080,
"protocol": "tcp"
}
],
"environment": [
{"name": "NODE_ENV", "value": "production"}
],
"secrets": [
{
"name": "DB_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:db-password"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/web-app",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs",
"mode": "non-blocking",
"max-buffer-size": "25m"
}
},
"healthCheck": {
"command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
"interval": 30,
"timeout": 5,
"retries": 3,
"startPeriod": 60
}
}
]
}
EOF
aws ecs register-task-definition --cli-input-json file://task-definition.json
Create Service with Load Balancer
aws ecs create-service \
--cluster my-cluster \
--service-name web-service \
--task-definition web-app:1 \
--desired-count 2 \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={
subnets=[subnet-12345678,subnet-87654321],
securityGroups=[sg-12345678],
assignPublicIp=DISABLED
}" \
--load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/web-tg/1234567890123456,containerName=web,containerPort=8080" \
--health-check-grace-period-seconds 60 \
--deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}"
Run Standalone Task
aws ecs run-task \
--cluster my-cluster \
--task-definition my-batch-job:1 \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={
subnets=[subnet-12345678],
securityGroups=[sg-12345678],
assignPublicIp=ENABLED
}"
Update Service (Deploy New Image)
# Register new task definition with updated image
aws ecs register-task-definition --cli-input-json file://task-definition.json
# Update service to use new version
aws ecs update-service \
--cluster my-cluster \
--service web-service \
--task-definition web-app:2 \
--force-new-deployment
Fargate Spot with SQS-Based Scaling
Use FARGATE_SPOT for batch/queue workloads to cut costs ~70%. Always include a fallback to regular FARGATE.
# Create service with Spot + fallback
aws ecs create-service \
--cluster batch-cluster \
--service-name queue-processor \
--task-definition my-processor:1 \
--desired-count 0 \
--capacity-provider-strategy \
capacityProvider=FARGATE_SPOT,weight=4,base=0 \
capacityProvider=FARGATE,weight=1,base=1 \
--network-configuration "awsvpcConfiguration={
subnets=[subnet-12345678],
securityGroups=[sg-12345678],
assignPublicIp=DISABLED
}"
# Register scalable target (scale to zero when queue empty)
aws application-autoscaling register-scalable-target \
--service-namespace ecs \
--resource-id service/batch-cluster/queue-processor \
--scalable-dimension ecs:service:DesiredCount \
--min-capacity 0 \
--max-capacity 20
# Scale-out alarm: messages > 100
aws cloudwatch put-metric-alarm \
--alarm-name queue-scale-out \
--metric-name ApproximateNumberOfMessagesVisible \
--namespace AWS/SQS \
--dimensions Name=QueueName,Value=my-queue \
--statistic Average \
--period 60 \
--evaluation-periods 1 \
--threshold 100 \
--comparison-operator GreaterThanThreshold \
--alarm-actions <scale-out-policy-arn>
# Scale-in alarm: queue empty for 3 periods (conservative to avoid flapping)
aws cloudwatch put-metric-alarm \
--alarm-name queue-scale-in \
--metric-name ApproximateNumberOfMessagesVisible \
--namespace AWS/SQS \
--dimensions Name=QueueName,Value=my-queue \
--statistic Average \
--period 60 \
--evaluation-periods 3 \
--threshold 0 \
--comparison-operator LessThanOrEqualToThreshold \
--alarm-actions <scale-in-policy-arn>
Backlog per task (AWS-recommended): Raw queue depth over-scales. Target-track queue depth / RunningTaskCount via metric math instead; Application Auto Scaling manages the alarms. Requires Container Insights on the cluster (RunningTaskCount in ECS/ContainerInsights). With 0 running tasks the divisor has no data, so this cannot scale from zero; keep the alarm pattern above when min-capacity is 0.
# TargetValue = acceptable latency / avg processing time per message (e.g. 10s / 0.1s = 100)
aws application-autoscaling put-scaling-policy \
--service-namespace ecs \
--resource-id service/batch-cluster/queue-processor \
--scalable-dimension ecs:service:DesiredCount \
--policy-name sqs-backlog-per-task \
--policy-type TargetTrackingScaling \
--target-tracking-scaling-policy-configuration '{
"TargetValue": 100,
"CustomizedMetricSpecification": {
"Metrics": [
{"Id": "m1", "ReturnData": false, "MetricStat": {"Stat": "Sum",
"Metric": {"Namespace": "AWS/SQS", "MetricName": "ApproximateNumberOfMessagesVisible",
"Dimensions": [{"Name": "QueueName", "Value": "my-queue"}]}}},
{"Id": "m2", "ReturnData": false, "MetricStat": {"Stat": "Average",
"Metric": {"Namespace": "ECS/ContainerInsights", "MetricName": "RunningTaskCount",
"Dimensions": [{"Name": "ClusterName", "Value": "batch-cluster"},
{"Name": "ServiceName", "Value": "queue-processor"}]}}},
{"Id": "e1", "Expression": "m1 / m2", "ReturnData": true}
]
}
}'
Protect in-flight work from scale-in: the worker sets ProtectionEnabled=true while processing (container agent endpoint $ECS_AGENT_URI/task-protection/v1/state, or aws ecs update-task-protection --cluster <c> --tasks <id> --protection-enabled --expires-in-minutes 60) and clears it when done. Task role needs ecs:GetTaskProtection and ecs:UpdateTaskProtection. Service tasks only.
Fargate Spot interruption handling: Spot tasks receive a SIGTERM 2 minutes before termination. Catch it in your application for graceful shutdown. For SQS consumers, call ChangeMessageVisibility on in-flight messages so they return to the queue rather than timing out.
Auto Scaling
# Register scalable target
aws application-autoscaling register-scalable-target \
--service-namespace ecs \
--resource-id service/my-cluster/web-service \
--scalable-dimension ecs:service:DesiredCount \
--min-capacity 2 \
--max-capacity 10
# Target tracking policy
aws application-autoscaling put-scaling-policy \
--service-namespace ecs \
--resource-id service/my-cluster/web-service \
--scalable-dimension ecs:service:DesiredCount \
--policy-name cpu-target-tracking \
--policy-type TargetTrackingScaling \
--target-tracking-scaling-policy-configuration '{
"TargetValue": 70.0,
"PredefinedMetricSpecification": {
"PredefinedMetricType": "ECSServiceAverageCPUUtilization"
},
"ScaleOutCooldown": 60,
"ScaleInCooldown": 120
}'
Faster scaling with 20-second metrics: enable high-resolution service metrics, then use ECSServiceAverageCPUUtilizationHighResolution or ECSServiceAverageMemoryUtilizationHighResolution as the PredefinedMetricType. On an existing service the --monitoring change triggers a deployment; create the high-res policy only after it completes. Not supported with CODE_DEPLOY/EXTERNAL deployment controllers. Extra CloudWatch charges apply.
aws ecs update-service \
--cluster my-cluster \
--service web-service \
--monitoring "metricConfigurations=[{metricNames=[CPUUtilization,MemoryUtilization],resolutionSeconds=20}]"
CLI Reference
Cluster Management
| Command | Description |
|---|---|
aws ecs create-cluster |
Create cluster |
aws ecs describe-clusters |
Get cluster details |
aws ecs list-clusters |
List clusters |
aws ecs delete-cluster |
Delete cluster |
Task Definitions
| Command | Description |
|---|---|
aws ecs register-task-definition |
Create task definition |
aws ecs describe-task-definition |
Get task definition |
aws ecs list-task-definitions |
List task definitions |
aws ecs deregister-task-definition |
Deregister version |
Services
| Command | Description |
|---|---|
aws ecs create-service |
Create service |
aws ecs update-service |
Update service |
aws ecs describe-services |
Get service details |
aws ecs delete-service |
Delete service |
Tasks
| Command | Description |
|---|---|
aws ecs run-task |
Run standalone task |
aws ecs stop-task |
Stop running task |
aws ecs describe-tasks |
Get task details |
aws ecs list-tasks |
List tasks |
Best Practices
Security
- Use task roles for AWS API access (not access keys)
- Use execution roles for ECR/Secrets access
- Store secrets in Secrets Manager or Parameter Store
- Use private subnets with NAT gateway
- Enable CloudTrail for API auditing
- Cap task size with IAM —
ecs:task-cpu/ecs:task-memorycondition keys apply toRunTaskandStartTaskas well asRegisterTaskDefinition,CreateService,UpdateService
Performance
- Right-size CPU/memory — monitor and adjust
- Use Fargate Spot for fault-tolerant workloads (70% savings)
- Enable container insights for monitoring
- Use service discovery for internal communication
Reliability
- Deploy across multiple AZs
- Configure health checks properly
- Set appropriate deregistration delay
- Use circuit breaker for deployments
aws ecs update-service \
--cluster my-cluster \
--service web-service \
--deployment-configuration '{
"deploymentCircuitBreaker": {
"enable": true,
"rollback": true,
"resetOnHealthyTask": true,
"thresholdConfiguration": {"type": "BOUNDED_PERCENT", "value": 50}
}
}'
- Tune circuit breaker threshold: default
BOUNDED_PERCENT/50 = 50% of desired count, clamped to 3-200 failures.UNBOUNDED_PERCENTdrops the clamp (large services);COUNTusesvalueas a fixed failure count (e.g. low for fast dev rollbacks).resetOnHealthyTask: falsecounts failures cumulatively instead of consecutively. - Early success criteria (rolling only): mark the deployment successful once
healthyPercentof desired tasks are healthy on the new revision; the rest launch via normal service scaling.healthyPercentmust be betweenminimumHealthyPercentand 100; replica services default tominimumHealthyPercent100, so set it explicitly when using a lowerhealthyPercent. After early completion, circuit breaker and alarm rollback no longer apply.sourceServiceRevisionCleanup:BLOCKINGdrains old tasks before success;DEFERREDdeclares success first and drains old tasks asynchronously (long-lived connections, scale-in protection).
aws ecs update-service \
--cluster my-cluster \
--service web-service \
--deployment-configuration '{
"strategy": "ROLLING",
"minimumHealthyPercent": 75,
"earlySuccessCriteria": {"enable": true, "healthyPercent": 90, "sourceServiceRevisionCleanup": "BLOCKING"}
}'
- Service Connect zone-aware routing is on by default (prefers same-AZ endpoints, cuts cross-AZ cost); existing services need one redeploy to pick it up.
- EC2 launch type: migrate to Amazon Linux 2023 ECS-optimized AMIs. AL2 ECS-optimized AMIs reached end of life June 30, 2026 (no new AMIs, agent pinned).
- Fargate: platform version
1.3.0was deprecated June 15, 2026. UseLATESTor1.4.0.
Cost Optimization
- Use Fargate Spot for batch workloads
- Right-size task resources
- Scale to zero when not needed
- Use capacity providers for mixed Fargate/Spot
Troubleshooting
Task Fails to Start
Check:
# View stopped tasks
aws ecs describe-tasks \
--cluster my-cluster \
--tasks $(aws ecs list-tasks --cluster my-cluster --desired-status STOPPED --query 'taskArns[0]' --output text)
Common causes:
- Image not found (ECR permissions)
- Secrets access denied
- Network configuration (subnets, security groups)
- Resource limits exceeded
Container Keeps Restarting
Debug:
# Check CloudWatch logs
aws logs get-log-events \
--log-group-name /ecs/web-app \
--log-stream-name "ecs/web/abc123"
# Check task details
aws ecs describe-tasks \
--cluster my-cluster \
--tasks task-arn \
--query 'tasks[0].containers[0].{reason:reason,exitCode:exitCode}'
Causes:
- Health check failing
- Application crashing
- Out of memory
Live Debugging with ECS Exec
Connect directly to a running container without SSH. Requires enableExecuteCommand: true on the service and the SSM agent in your container image (included in most base images).
# Enable on existing service
aws ecs update-service \
--cluster my-cluster \
--service web-service \
--enable-execute-command
# Get a shell in a running task
TASK_ARN=$(aws ecs list-tasks --cluster my-cluster --service-name web-service \
--query 'taskArns[0]' --output text)
aws ecs execute-command \
--cluster my-cluster \
--task $TASK_ARN \
--container web \
--interactive \
--command "/bin/sh"
Requirements: Task role must have ssmmessages:CreateControlChannel, ssmmessages:CreateDataChannel, ssmmessages:OpenControlChannel, ssmmessages:OpenDataChannel permissions.
Service Stuck Deploying
# Check deployment status
aws ecs describe-services \
--cluster my-cluster \
--services web-service \
--query 'services[0].deployments'
# Check events
aws ecs describe-services \
--cluster my-cluster \
--services web-service \
--query 'services[0].events[:5]'
Causes:
- Health check failing on new tasks
- Not enough capacity
- Target group health checks failing
Action Logs (What ECS Did During a Deployment)
Opt-in per cluster. Timestamped records of actions ECS takes during service deployments (state transitions, rollbacks, lifecycle hooks) and Managed Daemon lifecycle, with logLevel INFO/WARN/ERROR and status reasons. Keeps failure metadata past the 1-hour stopped-task retention. Billed as CloudWatch vended logs.
aws logs put-delivery-source \
--name my-ecs-action-logs \
--resource-arn arn:aws:ecs:us-east-1:123456789012:cluster/my-cluster \
--log-type EcsActionLogs
aws logs put-delivery-destination \
--name my-ecs-logs-destination \
--output-format json \
--delivery-destination-configuration '{"destinationResourceArn": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/vendedlogs/ecs/action-logs/my-cluster"}'
aws logs create-delivery \
--delivery-source-name my-ecs-action-logs \
--delivery-destination-arn arn:aws:logs:us-east-1:123456789012:delivery-destination:my-ecs-logs-destination
Requires: logs:PutDeliverySource, logs:PutDeliveryDestination, logs:CreateDelivery, logs:GetDelivery, ecs:AllowVendedLogDeliveryForResource; the log group resource policy must allow delivery.logs.amazonaws.com to logs:CreateLogStream/logs:PutLogEvents. One log stream per service/daemon ARN.
OOM Kills After Moving EC2 Hosts to AL2023
AL2023 uses cgroup v2: with only task-level memory, the container cannot see the limit, so JVMs and similar runtimes size heap from host memory. Set container-level memory equal to the task memory, or set ECS_PROPAGATE_TASK_MEMORY_LIMIT_CGROUPV2=true in /etc/ecs/ecs.config (agent 1.104.0+). Reported memory also includes page cache on cgroup v2, so utilization reads higher than on AL2.
Cannot Pull Image from ECR
Check execution role has:
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Resource": "*"
}
Also check:
- VPC endpoint for ECR (if private subnet)
- NAT gateway (if private subnet)
- Security group allows HTTPS outbound
References
Files (aws-agent-skills)
-
SKILL.md 18.3 KB
--- name: ecs description: AWS ECS container orchestration for running Docker containers. Use when deploying containerized applications, configuring task definitions, setting up services, managing clusters, or troubleshooting container issues. last_updated: "2026-09-14" doc_source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ --- # AWS ECS Amazon Elastic Container Service (ECS) is a fully managed container orchestration service. Run containers on AWS Fargate (serverless) or EC2 instances. ## Table of Contents - [Core Concepts](#core-concepts) - [Common Patterns](#common-patterns) - [CLI Reference](#cli-reference) - [Best Practices](#best-practices) - [Troubleshooting](#troubleshooting) - [References](#references) ## Core Concepts ### Cluster Logical grouping of tasks or services. Can contain Fargate tasks, EC2 instances, or both. ### Task Definition Blueprint for your application. Defines containers, resources, networking, and IAM roles. ### Task Running instance of a task definition. Can run standalone or as part of a service. ### Service Maintains desired count of tasks. Handles deployments, load balancing, and auto scaling. ### Launch Types | Type | Description | Use Case | |------|-------------|----------| | **Fargate** | Serverless, pay per task | Most workloads | | **EC2** | Self-managed instances | GPU, Windows, specific requirements | ## Common Patterns ### Create a Fargate Cluster **AWS CLI:** ```bash # Create cluster aws ecs create-cluster --cluster-name my-cluster # With capacity providers aws ecs create-cluster \ --cluster-name my-cluster \ --capacity-providers FARGATE FARGATE_SPOT \ --default-capacity-provider-strategy \ capacityProvider=FARGATE,weight=1 \ capacityProvider=FARGATE_SPOT,weight=1 ``` ### Register Task Definition ```bash cat > task-definition.json << 'EOF' { "family": "web-app", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "256", "memory": "512", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::123456789012:role/ecsTaskRole", "containerDefinitions": [ { "name": "web", "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest", "portMappings": [ { "containerPort": 8080, "protocol": "tcp" } ], "environment": [ {"name": "NODE_ENV", "value": "production"} ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:db-password" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/web-app", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs", "mode": "non-blocking", "max-buffer-size": "25m" } }, "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 60 } } ] } EOF aws ecs register-task-definition --cli-input-json file://task-definition.json ``` ### Create Service with Load Balancer ```bash aws ecs create-service \ --cluster my-cluster \ --service-name web-service \ --task-definition web-app:1 \ --desired-count 2 \ --launch-type FARGATE \ --network-configuration "awsvpcConfiguration={ subnets=[subnet-12345678,subnet-87654321], securityGroups=[sg-12345678], assignPublicIp=DISABLED }" \ --load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/web-tg/1234567890123456,containerName=web,containerPort=8080" \ --health-check-grace-period-seconds 60 \ --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}" ``` ### Run Standalone Task ```bash aws ecs run-task \ --cluster my-cluster \ --task-definition my-batch-job:1 \ --launch-type FARGATE \ --network-configuration "awsvpcConfiguration={ subnets=[subnet-12345678], securityGroups=[sg-12345678], assignPublicIp=ENABLED }" ``` ### Update Service (Deploy New Image) ```bash # Register new task definition with updated image aws ecs register-task-definition --cli-input-json file://task-definition.json # Update service to use new version aws ecs update-service \ --cluster my-cluster \ --service web-service \ --task-definition web-app:2 \ --force-new-deployment ``` ### Fargate Spot with SQS-Based Scaling Use `FARGATE_SPOT` for batch/queue workloads to cut costs ~70%. Always include a fallback to regular `FARGATE`. ```bash # Create service with Spot + fallback aws ecs create-service \ --cluster batch-cluster \ --service-name queue-processor \ --task-definition my-processor:1 \ --desired-count 0 \ --capacity-provider-strategy \ capacityProvider=FARGATE_SPOT,weight=4,base=0 \ capacityProvider=FARGATE,weight=1,base=1 \ --network-configuration "awsvpcConfiguration={ subnets=[subnet-12345678], securityGroups=[sg-12345678], assignPublicIp=DISABLED }" # Register scalable target (scale to zero when queue empty) aws application-autoscaling register-scalable-target \ --service-namespace ecs \ --resource-id service/batch-cluster/queue-processor \ --scalable-dimension ecs:service:DesiredCount \ --min-capacity 0 \ --max-capacity 20 # Scale-out alarm: messages > 100 aws cloudwatch put-metric-alarm \ --alarm-name queue-scale-out \ --metric-name ApproximateNumberOfMessagesVisible \ --namespace AWS/SQS \ --dimensions Name=QueueName,Value=my-queue \ --statistic Average \ --period 60 \ --evaluation-periods 1 \ --threshold 100 \ --comparison-operator GreaterThanThreshold \ --alarm-actions <scale-out-policy-arn> # Scale-in alarm: queue empty for 3 periods (conservative to avoid flapping) aws cloudwatch put-metric-alarm \ --alarm-name queue-scale-in \ --metric-name ApproximateNumberOfMessagesVisible \ --namespace AWS/SQS \ --dimensions Name=QueueName,Value=my-queue \ --statistic Average \ --period 60 \ --evaluation-periods 3 \ --threshold 0 \ --comparison-operator LessThanOrEqualToThreshold \ --alarm-actions <scale-in-policy-arn> ``` **Backlog per task (AWS-recommended):** Raw queue depth over-scales. Target-track `queue depth / RunningTaskCount` via metric math instead; Application Auto Scaling manages the alarms. Requires Container Insights on the cluster (`RunningTaskCount` in `ECS/ContainerInsights`). With 0 running tasks the divisor has no data, so this cannot scale *from* zero; keep the alarm pattern above when `min-capacity` is 0. ```bash # TargetValue = acceptable latency / avg processing time per message (e.g. 10s / 0.1s = 100) aws application-autoscaling put-scaling-policy \ --service-namespace ecs \ --resource-id service/batch-cluster/queue-processor \ --scalable-dimension ecs:service:DesiredCount \ --policy-name sqs-backlog-per-task \ --policy-type TargetTrackingScaling \ --target-tracking-scaling-policy-configuration '{ "TargetValue": 100, "CustomizedMetricSpecification": { "Metrics": [ {"Id": "m1", "ReturnData": false, "MetricStat": {"Stat": "Sum", "Metric": {"Namespace": "AWS/SQS", "MetricName": "ApproximateNumberOfMessagesVisible", "Dimensions": [{"Name": "QueueName", "Value": "my-queue"}]}}}, {"Id": "m2", "ReturnData": false, "MetricStat": {"Stat": "Average", "Metric": {"Namespace": "ECS/ContainerInsights", "MetricName": "RunningTaskCount", "Dimensions": [{"Name": "ClusterName", "Value": "batch-cluster"}, {"Name": "ServiceName", "Value": "queue-processor"}]}}}, {"Id": "e1", "Expression": "m1 / m2", "ReturnData": true} ] } }' ``` **Protect in-flight work from scale-in:** the worker sets `ProtectionEnabled=true` while processing (container agent endpoint `$ECS_AGENT_URI/task-protection/v1/state`, or `aws ecs update-task-protection --cluster <c> --tasks <id> --protection-enabled --expires-in-minutes 60`) and clears it when done. Task role needs `ecs:GetTaskProtection` and `ecs:UpdateTaskProtection`. Service tasks only. **Fargate Spot interruption handling:** Spot tasks receive a SIGTERM 2 minutes before termination. Catch it in your application for graceful shutdown. For SQS consumers, call `ChangeMessageVisibility` on in-flight messages so they return to the queue rather than timing out. ### Auto Scaling ```bash # Register scalable target aws application-autoscaling register-scalable-target \ --service-namespace ecs \ --resource-id service/my-cluster/web-service \ --scalable-dimension ecs:service:DesiredCount \ --min-capacity 2 \ --max-capacity 10 # Target tracking policy aws application-autoscaling put-scaling-policy \ --service-namespace ecs \ --resource-id service/my-cluster/web-service \ --scalable-dimension ecs:service:DesiredCount \ --policy-name cpu-target-tracking \ --policy-type TargetTrackingScaling \ --target-tracking-scaling-policy-configuration '{ "TargetValue": 70.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilization" }, "ScaleOutCooldown": 60, "ScaleInCooldown": 120 }' ``` **Faster scaling with 20-second metrics:** enable high-resolution service metrics, then use `ECSServiceAverageCPUUtilizationHighResolution` or `ECSServiceAverageMemoryUtilizationHighResolution` as the `PredefinedMetricType`. On an existing service the `--monitoring` change triggers a deployment; create the high-res policy only after it completes. Not supported with `CODE_DEPLOY`/`EXTERNAL` deployment controllers. Extra CloudWatch charges apply. ```bash aws ecs update-service \ --cluster my-cluster \ --service web-service \ --monitoring "metricConfigurations=[{metricNames=[CPUUtilization,MemoryUtilization],resolutionSeconds=20}]" ``` ## CLI Reference ### Cluster Management | Command | Description | |---------|-------------| | `aws ecs create-cluster` | Create cluster | | `aws ecs describe-clusters` | Get cluster details | | `aws ecs list-clusters` | List clusters | | `aws ecs delete-cluster` | Delete cluster | ### Task Definitions | Command | Description | |---------|-------------| | `aws ecs register-task-definition` | Create task definition | | `aws ecs describe-task-definition` | Get task definition | | `aws ecs list-task-definitions` | List task definitions | | `aws ecs deregister-task-definition` | Deregister version | ### Services | Command | Description | |---------|-------------| | `aws ecs create-service` | Create service | | `aws ecs update-service` | Update service | | `aws ecs describe-services` | Get service details | | `aws ecs delete-service` | Delete service | ### Tasks | Command | Description | |---------|-------------| | `aws ecs run-task` | Run standalone task | | `aws ecs stop-task` | Stop running task | | `aws ecs describe-tasks` | Get task details | | `aws ecs list-tasks` | List tasks | ## Best Practices ### Security - **Use task roles** for AWS API access (not access keys) - **Use execution roles** for ECR/Secrets access - **Store secrets in Secrets Manager** or Parameter Store - **Use private subnets** with NAT gateway - **Enable CloudTrail** for API auditing - **Cap task size with IAM** — `ecs:task-cpu` / `ecs:task-memory` condition keys apply to `RunTask` and `StartTask` as well as `RegisterTaskDefinition`, `CreateService`, `UpdateService` ### Performance - **Right-size CPU/memory** — monitor and adjust - **Use Fargate Spot** for fault-tolerant workloads (70% savings) - **Enable container insights** for monitoring - **Use service discovery** for internal communication ### Reliability - **Deploy across multiple AZs** - **Configure health checks** properly - **Set appropriate deregistration delay** - **Use circuit breaker** for deployments ```bash aws ecs update-service \ --cluster my-cluster \ --service web-service \ --deployment-configuration '{ "deploymentCircuitBreaker": { "enable": true, "rollback": true, "resetOnHealthyTask": true, "thresholdConfiguration": {"type": "BOUNDED_PERCENT", "value": 50} } }' ``` - **Tune circuit breaker threshold:** default `BOUNDED_PERCENT`/50 = 50% of desired count, clamped to 3-200 failures. `UNBOUNDED_PERCENT` drops the clamp (large services); `COUNT` uses `value` as a fixed failure count (e.g. low for fast dev rollbacks). `resetOnHealthyTask: false` counts failures cumulatively instead of consecutively. - **Early success criteria** (rolling only): mark the deployment successful once `healthyPercent` of desired tasks are healthy on the new revision; the rest launch via normal service scaling. `healthyPercent` must be between `minimumHealthyPercent` and 100; replica services default to `minimumHealthyPercent` 100, so set it explicitly when using a lower `healthyPercent`. After early completion, circuit breaker and alarm rollback no longer apply. `sourceServiceRevisionCleanup`: `BLOCKING` drains old tasks before success; `DEFERRED` declares success first and drains old tasks asynchronously (long-lived connections, scale-in protection). ```bash aws ecs update-service \ --cluster my-cluster \ --service web-service \ --deployment-configuration '{ "strategy": "ROLLING", "minimumHealthyPercent": 75, "earlySuccessCriteria": {"enable": true, "healthyPercent": 90, "sourceServiceRevisionCleanup": "BLOCKING"} }' ``` - **Service Connect zone-aware routing** is on by default (prefers same-AZ endpoints, cuts cross-AZ cost); existing services need one redeploy to pick it up. - **EC2 launch type: migrate to Amazon Linux 2023** ECS-optimized AMIs. AL2 ECS-optimized AMIs reached end of life June 30, 2026 (no new AMIs, agent pinned). - **Fargate:** platform version `1.3.0` was deprecated June 15, 2026. Use `LATEST` or `1.4.0`. ### Cost Optimization - **Use Fargate Spot** for batch workloads - **Right-size task resources** - **Scale to zero** when not needed - **Use capacity providers** for mixed Fargate/Spot ## Troubleshooting ### Task Fails to Start **Check:** ```bash # View stopped tasks aws ecs describe-tasks \ --cluster my-cluster \ --tasks $(aws ecs list-tasks --cluster my-cluster --desired-status STOPPED --query 'taskArns[0]' --output text) ``` **Common causes:** - Image not found (ECR permissions) - Secrets access denied - Network configuration (subnets, security groups) - Resource limits exceeded ### Container Keeps Restarting **Debug:** ```bash # Check CloudWatch logs aws logs get-log-events \ --log-group-name /ecs/web-app \ --log-stream-name "ecs/web/abc123" # Check task details aws ecs describe-tasks \ --cluster my-cluster \ --tasks task-arn \ --query 'tasks[0].containers[0].{reason:reason,exitCode:exitCode}' ``` **Causes:** - Health check failing - Application crashing - Out of memory ### Live Debugging with ECS Exec Connect directly to a running container without SSH. Requires `enableExecuteCommand: true` on the service and the SSM agent in your container image (included in most base images). ```bash # Enable on existing service aws ecs update-service \ --cluster my-cluster \ --service web-service \ --enable-execute-command # Get a shell in a running task TASK_ARN=$(aws ecs list-tasks --cluster my-cluster --service-name web-service \ --query 'taskArns[0]' --output text) aws ecs execute-command \ --cluster my-cluster \ --task $TASK_ARN \ --container web \ --interactive \ --command "/bin/sh" ``` **Requirements:** Task role must have `ssmmessages:CreateControlChannel`, `ssmmessages:CreateDataChannel`, `ssmmessages:OpenControlChannel`, `ssmmessages:OpenDataChannel` permissions. ### Service Stuck Deploying ```bash # Check deployment status aws ecs describe-services \ --cluster my-cluster \ --services web-service \ --query 'services[0].deployments' # Check events aws ecs describe-services \ --cluster my-cluster \ --services web-service \ --query 'services[0].events[:5]' ``` **Causes:** - Health check failing on new tasks - Not enough capacity - Target group health checks failing ### Action Logs (What ECS Did During a Deployment) Opt-in per cluster. Timestamped records of actions ECS takes during service deployments (state transitions, rollbacks, lifecycle hooks) and Managed Daemon lifecycle, with `logLevel` INFO/WARN/ERROR and status reasons. Keeps failure metadata past the 1-hour stopped-task retention. Billed as CloudWatch vended logs. ```bash aws logs put-delivery-source \ --name my-ecs-action-logs \ --resource-arn arn:aws:ecs:us-east-1:123456789012:cluster/my-cluster \ --log-type EcsActionLogs aws logs put-delivery-destination \ --name my-ecs-logs-destination \ --output-format json \ --delivery-destination-configuration '{"destinationResourceArn": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/vendedlogs/ecs/action-logs/my-cluster"}' aws logs create-delivery \ --delivery-source-name my-ecs-action-logs \ --delivery-destination-arn arn:aws:logs:us-east-1:123456789012:delivery-destination:my-ecs-logs-destination ``` **Requires:** `logs:PutDeliverySource`, `logs:PutDeliveryDestination`, `logs:CreateDelivery`, `logs:GetDelivery`, `ecs:AllowVendedLogDeliveryForResource`; the log group resource policy must allow `delivery.logs.amazonaws.com` to `logs:CreateLogStream`/`logs:PutLogEvents`. One log stream per service/daemon ARN. ### OOM Kills After Moving EC2 Hosts to AL2023 AL2023 uses cgroup v2: with only task-level `memory`, the container cannot see the limit, so JVMs and similar runtimes size heap from host memory. Set container-level `memory` equal to the task memory, or set `ECS_PROPAGATE_TASK_MEMORY_LIMIT_CGROUPV2=true` in `/etc/ecs/ecs.config` (agent 1.104.0+). Reported memory also includes page cache on cgroup v2, so utilization reads higher than on AL2. ### Cannot Pull Image from ECR **Check execution role has:** ```json { "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage" ], "Resource": "*" } ``` **Also check:** - VPC endpoint for ECR (if private subnet) - NAT gateway (if private subnet) - Security group allows HTTPS outbound ## References - [ECS Developer Guide](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/) - [ECS API Reference](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/) - [ECS CLI Reference](https://docs.aws.amazon.com/cli/latest/reference/ecs/) - [boto3 ECS](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html) -
task-definitions.md 9.2 KB
# ECS Task Definitions Detailed patterns for ECS task definitions. ## Task Definition Structure ```json { "family": "my-app", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "256", "memory": "512", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::123456789012:role/ecsTaskRole", "containerDefinitions": [...], "volumes": [...], "runtimePlatform": { "cpuArchitecture": "ARM64", "operatingSystemFamily": "LINUX" } } ``` ## CPU and Memory Combinations (Fargate) | CPU | Memory Options | |-----|----------------| | 256 (.25 vCPU) | 512 MB, 1 GB, 2 GB | | 512 (.5 vCPU) | 1-4 GB (1 GB increments) | | 1024 (1 vCPU) | 2-8 GB (1 GB increments) | | 2048 (2 vCPU) | 4-16 GB (1 GB increments) | | 4096 (4 vCPU) | 8-30 GB (1 GB increments) | | 8192 (8 vCPU) | 16-60 GB (4 GB increments) | | 16384 (16 vCPU) | 32-120 GB (8 GB increments) | | 32768 (32 vCPU) | 60 GB, 120 GB, 244 GB (Linux, x86 or ARM) | ## Container Definition Examples ### Web Application ```json { "name": "web", "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest", "essential": true, "portMappings": [ { "containerPort": 8080, "protocol": "tcp", "appProtocol": "http" } ], "environment": [ {"name": "NODE_ENV", "value": "production"}, {"name": "PORT", "value": "8080"} ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:db-password:password::" }, { "name": "API_KEY", "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/my-app/api-key" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/my-app", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "web" } }, "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 60 } } ``` ### Sidecar Pattern ```json { "containerDefinitions": [ { "name": "app", "image": "my-app:latest", "essential": true, "portMappings": [{"containerPort": 8080}], "dependsOn": [ {"containerName": "envoy", "condition": "HEALTHY"} ] }, { "name": "envoy", "image": "envoyproxy/envoy:v1.28.0", "essential": true, "portMappings": [{"containerPort": 9901}], "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:9901/ready || exit 1"], "interval": 5, "timeout": 2, "retries": 3, "startPeriod": 10 } }, { "name": "xray-daemon", "image": "amazon/aws-xray-daemon", "essential": false, "portMappings": [{"containerPort": 2000, "protocol": "udp"}], "memory": 256 } ] } ``` ### Init Container Pattern ```json { "containerDefinitions": [ { "name": "init-db", "image": "my-migrations:latest", "essential": false, "command": ["./run-migrations.sh"], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/my-app", "awslogs-stream-prefix": "init" } } }, { "name": "app", "image": "my-app:latest", "essential": true, "dependsOn": [ {"containerName": "init-db", "condition": "SUCCESS"} ] } ] } ``` ## Secrets Management ### From Secrets Manager ```json { "secrets": [ { "name": "FULL_SECRET", "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret" }, { "name": "SPECIFIC_KEY", "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret:username::" }, { "name": "SPECIFIC_VERSION", "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-secret:password::abc123" } ] } ``` ### From Parameter Store ```json { "secrets": [ { "name": "API_KEY", "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/my-app/api-key" }, { "name": "DB_HOST", "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/my-app/db-host" } ] } ``` ### Execution Role Permissions ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": [ "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-app/*" ] }, { "Effect": "Allow", "Action": [ "ssm:GetParameters" ], "Resource": [ "arn:aws:ssm:us-east-1:123456789012:parameter/my-app/*" ] }, { "Effect": "Allow", "Action": [ "kms:Decrypt" ], "Resource": [ "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012" ] } ] } ``` ## Logging Configurations ### CloudWatch Logs ```json { "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/my-app", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs", "awslogs-create-group": "true", "mode": "non-blocking", "max-buffer-size": "4m" } } } ``` ### FireLens (Fluent Bit) ```json { "containerDefinitions": [ { "name": "log-router", "image": "amazon/aws-for-fluent-bit:latest", "essential": true, "firelensConfiguration": { "type": "fluentbit", "options": { "enable-ecs-log-metadata": "true" } } }, { "name": "app", "image": "my-app:latest", "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "cloudwatch", "region": "us-east-1", "log_group_name": "/ecs/my-app", "log_stream_prefix": "app-", "auto_create_group": "true" } } } ] } ``` ## Volume Mounts ### EFS Volume ```json { "volumes": [ { "name": "shared-data", "efsVolumeConfiguration": { "fileSystemId": "fs-12345678", "rootDirectory": "/", "transitEncryption": "ENABLED", "authorizationConfig": { "accessPointId": "fsap-12345678", "iam": "ENABLED" } } } ], "containerDefinitions": [ { "name": "app", "mountPoints": [ { "sourceVolume": "shared-data", "containerPath": "/data", "readOnly": false } ] } ] } ``` ### Bind Mount (Fargate) ```json { "volumes": [ { "name": "scratch" } ], "containerDefinitions": [ { "name": "app", "mountPoints": [ { "sourceVolume": "scratch", "containerPath": "/tmp/scratch" } ] } ] } ``` ### tmpfs (Memory-Backed Scratch) Linux tasks on Fargate, Managed Instances, and EC2. Good for caches, short-lived secrets, and writable paths with `readonlyRootFilesystem`. Data is gone when the task stops. ```json { "containerDefinitions": [ { "name": "app", "readonlyRootFilesystem": true, "linuxParameters": { "tmpfs": [ {"containerPath": "/tmp", "size": 256, "mountOptions": ["noexec", "nosuid"]} ] } } ] } ``` `size` is MiB (required); `mountOptions` optional. ## Resource Limits ### Per-Container Limits ```json { "containerDefinitions": [ { "name": "app", "cpu": 512, "memory": 1024, "memoryReservation": 512 }, { "name": "sidecar", "cpu": 256, "memory": 512 } ] } ``` ### GPU (EC2 Launch Type) ```json { "requiresCompatibilities": ["EC2"], "containerDefinitions": [ { "name": "ml-training", "image": "my-ml-image:latest", "resourceRequirements": [ { "type": "GPU", "value": "1" } ] } ] } ``` ## Health Checks ### HTTP Health Check ```json { "healthCheck": { "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"], "interval": 30, "timeout": 5, "retries": 3, "startPeriod": 60 } } ``` ### TCP Health Check ```json { "healthCheck": { "command": ["CMD-SHELL", "nc -z localhost 8080 || exit 1"], "interval": 30, "timeout": 5, "retries": 3 } } ``` ### Script-Based Health Check ```json { "healthCheck": { "command": ["CMD", "/app/healthcheck.sh"], "interval": 30, "timeout": 10, "retries": 3, "startPeriod": 120 } } ``` ## Container Dependencies ```json { "containerDefinitions": [ { "name": "database", "essential": true, "healthCheck": {...} }, { "name": "cache", "essential": false, "dependsOn": [ {"containerName": "database", "condition": "HEALTHY"} ] }, { "name": "app", "essential": true, "dependsOn": [ {"containerName": "database", "condition": "HEALTHY"}, {"containerName": "cache", "condition": "START"} ] } ] } ``` Conditions: - `START`: Container has started - `COMPLETE`: Container ran to completion (any exit code) - `SUCCESS`: Container exited with code 0 - `HEALTHY`: Container health check passed
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.