How we mitigated CVE-2019-5736 for Unified VRM

tl;dr

  • CVE-2019-5736 is a runc vulnerability that allows attackers to obtain root access of any host running a docker container.
  • We mitigated CVE-2019-5736 on Tuesday, February 12th to increase the security of our platform.
  • Research and triage took about 2-3 hours. Applying the patch took about 15 minutes.

Architecture decisions

Containers are our infrastructure of choice because we found them easy, portable, and reliable. Containerization provides us with the assurance that our applications will run the same locally, in our testing environments, and in our production environments.

RunC is a universal container runtime used by docker. This impacted Unified VRM because we heavily use docker containers to run the platform. When we heard about the severity of this CVE, we jumped to mitigate it right away.

Our infrastructure is set up to address these vulnerabilities quickly. We primarily use AWS to host our infrastructure which also makes patching critical vulnerabilities like this a breeze.

AWS EC2 impact and mitigation

We were able to apply this patch with zero customer downtime by utilizing serverless architecture and rolling deployments for our EC2 instances.

The first piece of infrastructure that was affected by this CVE was our Elastic Beanstalk stack. Patching this was as simple as clicking a button to apply the latest platform update. These platform updates were performed in a rolling deployment style so there was always at least one operational container to serve incoming requests.

AWS ECS impact and mitigation

The next piece of our infrastructure we needed to update was our ECS fargate tasks. AWS released a patch for CVE-2019-5736 under fargate platform version 1.3. Our fargate services act as message processors for kicking off long running resource intensive tasks.

Updating this piece of infrastructure was as simple as re-deploying our fargate service under fargate platform version 1.3. All of our fargate tasks run using the latest platform version which means any new tasks that get kicked off automatically have the latest patches applied.

Patch validation

We validated that we are running on the latest AWS versions for the fargate platform and the docker platform for Elastic beanstalk. One could also validate by manually checking the actual versions of runc and docker to ensure they are upgraded.

Share your mitigation plans with us!

We love making vulnerability management easy and faster. We’d love to hear how you are mitigating this serious vulnerability.