Using ECS Exec with read-only root file system containers

As stated in the official ECS Exec documentation, ECS Exec today doesn’t support read-only containers.

The SSM agent requires that the container file system is able to be written to in order to create the required directories and files. Therefore, making the root file system read-only using the readonlyRootFilesystem task definition parameter, or any other method, isn’t supported.

This limitation means that you cannot use the readonlyRootFilesystem ECS task definition parameter in your tasks for example. When you validate your ECS cluster and task configuration with the amazon-ecs-exec-checker script¹, you’ll see the error as shown in the cover image above.

While ECS Exec (or the SSM agent more specifically) doesn’t support it for the rationale reason, some users want to use ECS Exec with read-only container storage for good reasons².

Note that because the ways I’ll show you here are just “workarounds” and of course they’re not officially supported by AWS, the workarounds could easily stop working by any changes made in Amazon ECS, or in the SSM agent.

TL;DR;

In summary, the workarounds I’m going to describe in this article are basically do the same thing - creating two explicitly writable directories, more specifically /var/lib/amazon and /var/log/amazon, for the SSM agent.

So make sure that you at least need to allow the agent to write to those directories, and the workarounds may not suit for you if you want to keep having fully read-only containers.

Implementing workaround

You can choose one of two ways to create writable directories:

1. Define volumes in Dockerfile
2. Define volumes in ECS task definition

1. Define volumes in Dockerfile

Here is an example of Dockerfile. As you can see, the point is line 3-5 and 7-9. (The base image “alpine:3.13” does not matter here of course).

2. Define volumes in ECS task definition

This is a bit complicated than the previous Dockerfile example, but you don’t have to modify your existing Dockerfile or container images if you take this approach.

Here is an excerpt of an example ECS task definition. The points are:

• Line 2-5: Define Bind mount volumes
• Line 12-23: Mount the defined volumes

Test it in your AWS account

You can create an ECS task definition using the “workarounds-included” CloudFormation template below to test in your AWS account.

There are four containers defined in the task definition for simplicity:

(bold texts are the container names)

1. disabled - The readonlyRootFilsystem option is disabled for this container. So you can exec into this container without any workaround (of course.)
2. enabled - The readonlyRootFilsystem option is enabled for this container and there’s no workaround in it. So you cannot exec into this container. (The SSM agent won’t run for this container.)
3. enabled-but-with-image-level-workaround - The readonlyRootFilsystem option is enabled but the container image has the Dockerfile-level workaround. You’ll be able to exec into this container.
4. enabled-but-with-task-def-level-workaround - The readonlyRootFilsystem option is enabled but the container has the ECS task definition level workaround. You’ll be able to exec into this container.