terraform - Amazon ECS on EC2 Server instance hitting RESOURCE:ENI error

Question

I am running ECS tasks (using awsvpc networking) on a "t3.small" EC2 server instance but am encountering RESOURCE:ENI errors when attempting to run more than 2 tasks, described here as:

For RESOURCE:ENI errors, your cluster does not have any available elastic network interface attachment points, which are required for tasks that use the awsvpc network mode. Amazon EC2 instances have a limit to the number of network interfaces that can be attached to them, and the primary network interface counts as one. For more information about how many network interfaces are supported per instance type, see IP Addresses Per Network Interface Per Instance Type in the Amazon EC2 User Guide for Linux Instances.

t3.small instance types are documented as supporting 4 private IPv4 addresses per interface:

My EC2 instance has a public IP address (while my ECS tasks have private IP addresses) which I think explains why I can only run 2 tasks, not 3. (Amazon states that the primary network interface counts as one.)

However, "Private Ipv4 addresses per interface" would seem to indicate that by assigning another network interface to my EC2 instance I could allow another 4 ECS tasks to run before hitting the RESOURCE:ENI error - but when I do this, I can only run 1 task! The additional interface apparently consumes one of the elastic network interface attachment points.

This doesn't seem right. I'm relatively new to docker and ECS, but a limit of 2 or 3 tasks/containers on a t3.small EC2 instance seems low. I thought it was good practice to use lots of small containers rather than a few big ones.

Are my only options to use larger instances or switch to bridge networking (even though Amazon recommend awsvpc), or am I missing something?

---

Updated to add: even when associating my EC2 server with private subnets, and assigning only its single default network interface, I am still apparently only able to run 2 tasks before hitting RESOURCE:ENI.

question from:https://stackoverflow.com/questions/66066448/amazon-ecs-on-ec2-server-instance-hitting-resourceeni-error

Answer 1

I misunderstood Amazon's documentation. "Private Ipv4 addresses per interface" is not related to the ENI limit for an EC2 instance.

aws ec2 describe-instance-types --filters Name=instance-type,Values=t3.* --query "InstanceTypes[].{Type: InstanceType, MaxENI: N
etworkInfo.MaximumNetworkInterfaces, IPv4addr: NetworkInfo.Ipv4AddressesPerInterface}" --output table
--------------------------------------
|        DescribeInstanceTypes       |
+----------+----------+--------------+
| IPv4addr | MaxENI   |    Type      |
+----------+----------+--------------+
|  12      |  3       |  t3.large    |
|  15      |  4       |  t3.2xlarge  |
|  6       |  3       |  t3.medium   |
|  15      |  4       |  t3.xlarge   |
|  4       |  3       |  t3.small    |
|  2       |  2       |  t3.micro    |
|  2       |  2       |  t3.nano     |
+----------+----------+--------------+

It is the MaxENI column here (or the 'Maximum Network Interfaces') that is the limit.

In other words, my t3.small instance (and the t3.medium and t3.large instances that I subsequently tested) all have a MaxENI of 3, so can only run 2 containers with awsvpc networking. A t3.xlarge (4vCPU, 16GiB RAM) instance can run 3 containers (also confirmed).

Amazon have a feature called Elastic network interface trunking which can raise this limit, described here. The introductory paragraph states:

For example, by default a c5.large instance may have up to three ENIs attached to it. The primary network interface for the instance counts as one, so you can attach an additional two ENIs to the instance. Because each task using the awsvpc network mode requires an ENI, you can typically only run two such tasks on this instance type.

... exactly my problem. The page continues:

Amazon ECS supports launching container instances with increased ENI density using supported Amazon EC2 instance types. When you use these instance types and opt in to the awsvpcTrunking account setting, additional ENIs are available on newly launched container instances. This configuration allows you to place more tasks using the awsvpc network mode on each container instance. Using this feature, a c5.large instance with awsvpcTrunking enabled has an increased ENI limit of twelve.