Using ECS Exec with read-only root file system containers
As stated in the official ECS Exec documentation, ECS Exec today doesn’t support read-only containers.
The SSM agent requires that the container file system is able to be written to in order to create the required directories and files. Therefore, making the root file system read-only using the readonlyRootFilesystem task definition parameter, or any other method, isn’t supported.
This limitation means that you cannot use the
readonlyRootFilesystem ECS task definition parameter in your tasks for example. When you validate your ECS cluster and task configuration with the
amazon-ecs-exec-checker script1, you’ll see the error as shown in the cover image above.
While ECS Exec (or the SSM agent more specifically) doesn’t support it for the rationale reason, some users want to use ECS Exec with read-only container storage for good reasons2.
Note that because the ways I’ll show you here are just “workarounds” and of course they’re not officially supported by AWS, the workarounds could easily stop working by any changes made in Amazon ECS, or in the SSM agent.
In summary, the workarounds I’m going to describe in this article are basically do the same thing - creating two explicitly writable directories, more specifically
/var/log/amazon, for the SSM agent.
So make sure that you at least need to allow the agent to write to those directories, and the workarounds may not suit for you if you want to keep having fully read-only containers.
You can choose one of two ways to create writable directories:
- Define volumes in
- Define volumes in ECS task definition
1. Define volumes in
Here is an example of Dockerfile. As you can see, the point is line 3-5 and 7-9. (The base image “alpine:3.13” does not matter here of course).
2. Define volumes in ECS task definition
This is a bit complicated than the previous Dockerfile example, but you don’t have to modify your existing Dockerfile or container images if you take this approach.
Here is an excerpt of an example ECS task definition. The points are:
- Line 2-5: Define Bind mount volumes
- Line 12-23: Mount the defined volumes
Test it in your AWS account
You can create an ECS task definition using the “workarounds-included” CloudFormation template below to test in your AWS account.
There are four containers defined in the task definition for simplicity:
(bold texts are the container names)
- disabled - The
readonlyRootFilsystemoption is disabled for this container. So you can exec into this container without any workaround (of course.)
- enabled - The
readonlyRootFilsystemoption is enabled for this container and there’s no workaround in it. So you cannot exec into this container. (The SSM agent won’t run for this container.)
- enabled-but-with-image-level-workaround - The
readonlyRootFilsystemoption is enabled but the container image has the
Dockerfile-level workaround. You’ll be able to exec into this container.
- enabled-but-with-task-def-level-workaround - The
readonlyRootFilsystemoption is enabled but the container has the ECS task definition level workaround. You’ll be able to exec into this container.