Using ECS Exec with read-only root file system containers

UPDATED 2024-08-31: Make the workaround work for the Fargate Platform Version 1.4.

As stated in the official ECS Exec documentation, ECS Exec today doesn’t support read-only containers.

The SSM agent requires that the container file system is able to be written to in order to create the required directories and files. Therefore, making the root file system read-only using the readonlyRootFilesystem task definition parameter, or any other method, isn’t supported.

This limitation means that you cannot use the readonlyRootFilesystem ECS task definition parameter enabled in your ECS tasks, so the amazon-ecs-exec-checker¹ prints errors for this configuration enabled as shown in the cover image above.

While ECS Exec (or the SSM agent more specifically) doesn’t support it for the rationale reason, some users want to use ECS Exec with read-only container storage for good reasons².

Note that because the ways I’ll show you below are just “workarounds” and of course they’re not officially supported by AWS, the workarounds could easily stop working by any changes made in Amazon ECS or in the SSM agent in the future.

TL;DR;

In summary, the workarounds I’m going to describe in this article are basically do the same thing - creating two explicitly writable directories, more specifically /var/lib/amazon and /var/log/amazon, for the SSM agent.

So make sure that you at least need to allow the agent to write to those directories, and the workarounds may not suit for you if you want to keep having fully read-only containers.

Implementing workaround

You can choose one of the two ways below to create writable directories
Starting from the Fargate Platform Version 1.4 (as explained in this GitHub issue comment³), the following steps are both required to make the workaround work:

1. Define volumes in Dockerfile
2. Define volumes in ECS task definition

1. Define volumes in Dockerfile

Here is an example of Dockerfile. See the line 3-5 and 7-9. (The base image alpine:3.13 does not matter here of course).

2. Define volumes in ECS task definition

This is a bit complicated than the previous Dockerfile example, but you don’t have to modify your existing Dockerfile or container images if you take this approach.

Here is an excerpt of an example ECS task definition. The important lines are:

• Line 2-5: Define Bind mount volumes
• Line 12-23: Mount the defined volumes

Test it in your AWS account

You can create an ECS task definition using the “workarounds-included” CloudFormation template below to test in your AWS account.

There are three containers defined in the task definition for simplicity:

(bold texts are the container names)

1. disabled - The readonlyRootFilsystem option is disabled for this container. So you can exec into this container without any workaround (of course.)
2. enabled - The readonlyRootFilsystem option is enabled for this container and there’s no workaround in it. So you cannot exec into this container. (The SSM agent won’t run for this container.)
3. enabled-with-workaround - The readonlyRootFilsystem option is enabled and the workaround is implemented. You’ll be able to exec into this container.