Accelerating dev container startup with automatic image caching

Wouter Verlaek / Senior Software Engineer at Gitpod / Jun 10, 2025

Environment startup took longer than it should. We had to rebuild dev containers from scratch every time, even for identical configurations.

We built automated dev container image caching that speeds up build times from minutes to seconds, without changing your existing configurations.

The Standard Approach: Manual CI-Based Prebuilds

The dev container ecosystem has recognized this challenge. The official dev containers documentation describes a recommended prebuild process where teams build and push container images to registries through CI pipelines. This approach works, but requires explicit workflow changes and coordination:

1. Set up CI pipelines to periodically build dev container images
2. Built images must be explicitly pushed to registries
3. Dev container configurations need manual updates to reference prebuilt images
4. Teams must coordinate when configuration changes invalidate existing prebuilds

While effective for some workflows, this manual approach creates operational overhead and requires teams to maintain additional CI infrastructure and coordination processes.

A Better Approach: Transparent Automation

But what if caching just happened automatically? What if your dev containers could start faster without you having to change anything?

That’s what we built. Gitpod’s AWS runners now include automatic dev container image caching. It works behind the scenes to speed up environment startup.

Performance Impact: Real-World Results

The performance improvements are significant. Here’s data from production environments across different project types:

The time savings are biggest for environments with complex dependency installations, multiple dev container features, or custom build steps.

Automatic Image Caching Strategy

When you create a new environment, we automatically check if an image has already been built:

We first check if a prebuilt image exists in AWS ECR for your exact devcontainer configuration. If found, we use secure, project-scoped credentials to pull the cached image, and your environment starts in seconds instead of minutes.

When no cached image exists, we build the container from scratch using devcontainer build, tag it with metadata (build duration, environment ID, creator), and push it to ECR. All subsequent environments from you and your team members automatically benefit from this cached image when using identical configurations.

This cuts startup time significantly while making sure everyone on the team benefits from faster environments without any coordination overhead.

How Dev Container Configuration is Preserved

When the system builds a new devcontainer image, it preserves additional configuration adjacent to the build such as feature volume mounts and lifecycle commands, and applies this configuration when creating a new dev container from the image.

Image Metadata Embedding

The devcontainer build process embeds your configuration into the image as metadata labels. The dev container specification describes which properties will be stored in the devcontainer.metadata container image label:

Preserved Configuration Elements:

• Features Metadata: Information about installed dev container features, including their mount requirements, environment variables, and lifecycle hooks
• Port Configurations: forwardPorts, portsAttributes, and port forwarding rules
• Environment Settings: containerEnv, remoteEnv, and other environment configurations
• User Configuration: remoteUser, containerUser, and permission settings
• Lifecycle Commands: onCreateCommand, updateContentCommand, postCreateCommand hooks
• Security Context: capAdd, securityOpt, privileged flags, and mount specifications

Configuration Merging During Startup

When starting an environment from a cached image, the devcontainer CLI merges the embedded metadata with your current devcontainer.json:

1. Read Image Labels: Extract the devcontainer.metadata label containing preserved configuration snippets
2. Merge Configurations: Combine the image metadata with the current devcontainer.json file
3. Apply Complete Context: Ensure features maintain their full functionality, including mounts, environment variables, and lifecycle hooks
4. Start Container: Launch with the complete merged configuration

This ensures that dev container features like Docker-in-Docker, language-specific tools, or custom mount configurations work the same whether you’re starting from a cached image or building fresh. The cached image contains not just the installed software, but all the configuration details needed to recreate your development environment.

User-Specific Configuration: Applied at Container Startup

We separate shared project configuration from user-specific personalization. This separation matters for both security and functionality.

What Gets Cached (Shared Project Elements):

• Base container image and installed packages
• Dev container features and their configurations
• Project-specific tools and dependencies
• Build-time environment variables and settings
• Lifecycle commands that prepare the environment for development

What Gets Applied Later (User-Specific Elements):

• Dotfiles: Personal shell configurations, aliases, and editor preferences are applied during devcontainer up using the dotfiles property, not during image build
• User Secrets: Environment variables containing sensitive information (API keys, tokens) are injected at container startup, never baked into cached images
• User-Scoped Commands: postCreateCommand and other lifecycle hooks that require user-specific context run after container creation
• Personal Tool Preferences: IDE extensions, themes, and settings that vary by developer

This setup gives us:

1. Security: Sensitive user data never leaks into shared cached images
2. Personalization: Each developer gets their personal development environment setup
3. Team Efficiency: The time-consuming project setup is shared while personal preferences stay individual
4. Cache Effectiveness: Images can be safely shared across team members without exposing personal configuration

When you start an environment from a cached image, we first set up the container with the shared project configuration, then apply your personal dotfiles and secrets. You get the speed benefits of caching while keeping the security and personalization you expect.

How It Works: Hash-Based Cache Invalidation

We compute a hash from your complete dev container configuration: contents of devcontainer.json, any referenced Dockerfile, and the relative path to the configuration file. This hash becomes the image tag in AWS ECR, enabling instant cache lookups. When any part of your configuration changes, the hash changes, automatically triggering a fresh build while keeping cached images for unchanged configurations.

When we find a cache hit, we dynamically override your dev container configuration to use the prebuilt image, removing build steps entirely while preserving all other configuration aspects.

Starting from a Prebuilt Image: Configuration Merging

When a cached image is available, we need to modify your devcontainer.json configuration to use the prebuilt image instead of triggering a new build. This happens through the devcontainer CLI’s --override-config flag.

We create an override configuration that preserves your original settings but removes build-related keys:

go

// Remove build-related keys to prevent triggering a new build
keysToRemove := []string{
    "dockerfile", "build", "features", "image",
    // Remove lifecycle hooks to prevent them running twice
    // (they're already baked into the prebuilt image)
    "onCreateCommand", "updateContentCommand", "postCreateCommand",
    "postStartCommand", "postAttachCommand",
}

// Point to the cached image instead
overrideConfig["image"] = imageName

When we run devcontainer up --override-config override-config.json, the CLI merges your original devcontainer.json with our override configuration. This merged result contains:

• Your original configuration: Port forwarding, environment variables, mounts, user settings
• Our image override: The prebuilt image reference instead of build instructions
• No duplicate lifecycle commands: Prevents commands from running twice since they’re already executed in the cached image

This approach ensures the container starts from your cached image but maintains all the runtime behavior you expect from your original configuration.

Security: Project-Scoped Access Control

Image caching needs careful security. We use a multi-layered approach that ensures strict project isolation:

Credential Generation and Scoping

The runner service generates temporary, scoped AWS credentials:

1. Role Assumption: The runner assumes a dedicated ECR access role using sts:AssumeRole
2. Session Tagging: Each role session is tagged with project-specific metadata:
   • gitpod.dev/project-id: Restricts access to specific project repositories
   • gitpod.dev/creator-id: Links to the environment creator
   • gitpod.dev/push: Controls whether push operations are allowed (only for initial builds)
3. Credential Handoff: Generated ECR auth tokens are passed to environments as encoded credentials
4. Immediate Cleanup: Push credentials are removed from environments after first use

IAM Policy Enforcement

We use both identity-based and resource-based AWS IAM policies for comprehensive access controls. The interesting part is how we use aws:PrincipalTag conditions with project-specific session tags to ensure environments can only access ECR repositories for their specific project.

The ECR access role includes identity-based policies that evaluate the session tags set during role assumption:

json

{
	"Sid": "AllowPullFromProject",
	"Effect": "Allow",
	"Action": [
		"ecr:BatchGetImage",
		"ecr:GetDownloadUrlForLayer",
		"ecr:DescribeImages"
	],
	"Resource": "arn:aws:ecr:region:account:repository/gitpod-runner-*/projects/${aws:PrincipalTag/gitpod.dev/project-id}/image-build"
}

The ${aws:PrincipalTag/gitpod.dev/project-id} variable dynamically resolves to the project ID tag set during role assumption, ensuring environments can only access repositories for their specific project.

For push operations, we add an additional layer of control using the gitpod.dev/push session tag:

json

{
	"Sid": "AllowPushFromProject",
	"Effect": "Allow",
	"Action": [
		"ecr:PutImage",
		"ecr:InitiateLayerUpload",
		"ecr:UploadLayerPart",
		"ecr:CompleteLayerUpload"
	],
	"Resource": "arn:aws:ecr:region:account:repository/gitpod-runner-*/projects/${aws:PrincipalTag/gitpod.dev/project-id}/image-build",
	"Condition": {
		"StringEquals": {
			"aws:PrincipalTag/gitpod.dev/push": "true"
		}
	}
}

This push tag is only set during initial environment builds, not for subsequent environments that only need to pull cached images. Each ECR repository also includes resource-based policies that provide defense-in-depth by independently verifying project access rights.

This approach ensures:

• Project isolation: Teams can only access images from their own projects
• Limited push access: Only initial environment builds can cache new images
• Credential separation: Push credentials are removed after use, subsequent rebuilds only pull
• Immutable storage: ECR immutability prevents tampering with cached images

Getting Started

For new AWS runners: Dev container image caching is enabled by default.

For existing AWS runners:

1. Update your CloudFormation stack to the latest version
2. Navigate to Settings → Runners in your Gitpod organization
3. Enable “Dev container image cache” for your AWS runner

The feature is available across all AWS regions and works with any dev container configuration that doesn’t use Docker Compose.

Implementation Details: Lifecycle Management

The cache includes automatic lifecycle management to balance performance with storage costs:

Image Metadata and Tracking

Every cached image includes metadata labels, to track the source that created the image, and allows us to track and display build time savings:

• Build Performance: gitpod.dev/build-duration stores the original build time, allowing the system to display accurate time savings when pulling cached images
• Build Attribution: gitpod.dev/environment-id and gitpod.dev/creator-id track which environment and developer created the cached image
• Build Timestamp: gitpod.dev/build-time enables lifecycle policies and debugging
• System Version: gitpod.dev/supervisor-version ensures compatibility across platform updates

When an environment successfully pulls a cached image, we read these labels to display meaningful feedback:

⚡️ Using pre-built dev container (saved ~4m45s build time)

This helps developers understand the performance benefits they’re getting.

Repository Structure and Naming

Images are organized in ECR repositories following a clear hierarchy:

gitpod-runner-{runnerID}/projects/{projectID}/image-build:{configHash}

This structure ensures runner and project isolation while enabling efficient cache lookups based on devcontainer configuration hashes.

Images automatically expire after 30 days, ensuring teams periodically benefit from security updates and dependency refreshes while preventing unlimited storage growth.

Beyond Performance: Strategic Benefits

Beyond faster startup times, automated caching enables broader improvements:

Reduced Context Switching Costs: When environment startup drops from 5 minutes to 30 seconds, developers are more likely to create fresh environments for each task, improving security and reducing configuration drift.

Faster Onboarding: New team members or external contributors can start contributing immediately without waiting for complex build processes.

Ephemeral Environment Adoption: Teams become more comfortable using short-lived environments when the startup cost is minimal, leading to better security practices and reduced infrastructure sprawl.

AI Agent Performance: AI agents that spin up fresh environments for each task benefit directly from faster startup times. Whether running automated workflows or handling parallel engineering tasks, faster environment creation means faster agent execution.

Comparison: Manual vs. Automated Approaches

The automated approach eliminates operational overhead while providing better security and lifecycle management.

Current Limitations and Future Directions

Our current implementation focuses on standard dev container configurations. Docker Compose-based dev containers aren’t supported yet due to limitations in the dev container CLI’s compose handling. The CLI does not support pre-building compose-based dev container images well, and specifically being able to start a compose-based setup from a prebuilt image transparently without updating the configuration on disk.

Wrapping Up

Automated dev container image caching eliminates the coordination overhead of manual prebuilds while providing better security and performance. It’s available for AWS runners today.

If this work is interesting to you, we are hiring Engineers in London, New York and Munich, see our open job roles here.