Infrastructure Update Changelog

This update might incur a downtime of up to 15 minutes due to EKS cluster maintenance and redis updates. We recommend doing this outside of working hours.

How to update

Follow the process laid out on Updating the Enterprise Infrastructure

Changelog

• Use encryption in transit with S3
• Upgrade Redis to version 7.1 and configure that minor version upgrades are automatically performed by AWS
• Upgrade EKS clusters to 1.28
• Increase retention period for CloudWatch Log Groups to two weeks, thereby making it easier to debug issues that occured further in the past

Changes to Gitpod Instance CF template

This is an optional infrastructure upgrade releasing a bug fix. You can choose to skip this release if the improvements listed are not urgently needed. This upgrade does not incur a downtime.

How to update

Changesets have been pre-created in your AWS Account. Your Gitpod Account Manager will provide you with links to the changeset for the role and the infrastructure. To apply the upgrade, you will need to visit these individually and execute the pre-generated changeset. Ensure to do this in the following order: first the role changeset, then the infrastructure changeset. Follow the process laid out on Updating the Enterprise Infrastructure. If you need access to the full, updated CloudFormation template, please contact your Gitpod Account Manager.

Changelog

• Cleanup the default security group of the Gitpod cell to not allow any traffic
• Fixes some policy requirements for a AWS Lambda role
• Your workspace can now access resources via a proxy. Please contact your Gitpod account manager to know how to enable these.
• Clean up of an unused log group

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

This update does not incurr downtime. Running workspaces are not affected. For extra caution, it is advised to perform the update outside of working hours.

How to update

Changesets have been pre-created in your AWS Account. Your Gitpod Account Manager will provide you with links to the changeset for the role and the infrastructure. To apply the upgrade, you will need to visit these individually and execute the pre-generated changeset. Ensure to do this in the following order: first the role changeset, then the infrastructure changeset. Follow the process laid out on Updating the Enterprise Infrastructure. If you need access to the full, updated CloudFormation template, please contact your Gitpod Account Manager.

Changelog

• HTTP2Preferred is now set on the HTTPS listeners for the workspace and meta load balancers. Results in better browser performance and reduces the amount of stale HTTP connections
• Support nameservers for allPrivate networking mode when using BYO Hosted Zone
• Enable providing only one hosted zone
• Support for proxy settings in workspace pods
• Routine version update for EKS addons

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

This update might incur a downtime of upto 5 minutes due to EKS cluster maintenance. We recommend doing this outside of working hours.

How to update

Changesets have been pre-created in your AWS Account. Your Gitpod Account Manager will provide you with links to the changeset for the role and the infrastructure. To apply the upgrade, you will need to visit these individually and execute the pre-generated changeset. Ensure to do this in the following order: first the role changeset, then the infrastructure changeset. Follow the process laid out on Updating the Enterprise Infrastructure. If you need access to the full, updated CloudFormation template, please contact your Gitpod Account Manager.

Changelog

• Introduce Container Image Signing for all images used by the dedicated cell
• Add support for Bringing Your Own(BYO) VPC and Hosted Zone to install Enterprise in
• Upgrade kube-proxy and vpc-cni cluster addons
• Add Role ARN to the output of the role template
• Fix AMI update workflow to cleanup all older nodes
• CloudFormation stack deletion now also cleans up the loadbalancer logs
• Fix policy issues in the pods in workspace cluster

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

This update does not incur any downtime.

How to update

Changesets have been pre-created in your AWS Account. Your Gitpod Account Manager will provide you with links to the changeset for the role and the infrastructure. To apply the upgrade, you will need to visit these individually and execute the pre-generated changeset. Ensure to do this in the following order: first the role changeset, then the infrastructure changeset. Follow the process laid out on Updating the Enterprise Infrastructure. If you need access to the full, updated CloudFormation template, please contact your Gitpod Account Manager.

Changelog

• Fix autoscaler configuration to use instance’s local timezone
• RDS access has been limited to the meta subnet, away from internal subnets
• CloudFormation stack deletion now fully deletes the instance without additional user input
• Remove duplicate entry for services DNS entry for public API gateway
• Remove inbound traffic traffic rules to the lambda security group
• Add Transit Gateway attachment ID and VPC ID to the CloudFormation stack output

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

This update transitions MySQL from v5.7 to v8 and incurs full downtime of up to 30 minutes.

How to update

This update uses a new update process that leverages pre-generated CloudFormation changesets that you only need to execute.

Your Gitpod Account Manager will provide you with links to the changeset for the role and the infrastructure. To apply the upgrade, you will need to visit these individually and execute the pre-generated changeset. Ensure to do this in the following order: first the role changeset, then the infrastructure changeset. Follow the process laid out on Updating the Enterprise Infrastructure. If you need access to the full, updated CloudFormation template, please contact your Gitpod Account Manager.

Changelog

• Major MySQL upgrade to v8
• Switches CA to rds-ca-rsa2048-g1 to avoid expiry in Aug 2024
• Policy to enable lambda updates
• Policies to support cleaner stack deletion

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

This update impacts running workspaces and should not be done during working hours. You can expect a downtime of 4 minutes after the CloudFormation Change Set is applied as new nodes are spun up.

How to update

• We have improved the template distribution process starting with this release. We now distribute templates as S3 URLs, which are readable only from the cell AWS account. This paves the way for a much more simplified upgrade process moving forward, so stay tuned!

• Your Gitpod Account Manager will provide you with two updated CloudFormation templates in the form of S3 URLs(one for the infrastructure template role and one for Gitpod itself) that both need to be applied as change sets.

• Follow the process laid out on Updating the Enterprise Infrastructure

Changelog

• Support for in-place AMI updates for all clusters
• Improved cell lambda image update workflow, triggered by new Enterprise releases. This equips a faster rollout of cell Lambda code changes. This does not change the permission scope of the lambdas.
• Automated changeset creation process, thereby simplifying future infrastructure upgrades. After this upgrade you will be able to preview all the changes automatically in the stack’s Changesets section
• Dynamic autoscaler configurations enabling fine tuning of cluster scaling capacity to optimally accommodate varying workloads. You can talk to your Gitpod Account manager to get this configured
• Export of historical logs, thereby enhancing the ease of debugging issues that customers encounter
• Various bug fixes

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

Changes to Gitpod Role CF template

Changes to Gitpod Instance CF template

This update impacts running workspaces and should not be done during working hours. You can expect a downtime of 5 minutes after the CloudFormation Change Set is applied as new nodes are spun up.

Creating the change set can take longer than usual. Further, once the change stack is applied, the clean up step will take longer than usual - up to 40 minutes (see below for reasoning). The Gitpod instance can be used as normal during this time. Future updates will take less time again.

How to update

• Your Gitpod Account Manager will provide you with two CloudFormation templates (one for the infrastructure template role and one for Gitpod itself) that both need to be applied as change sets.

• Follow the process laid out on Updating the Enterprise Infrastructure

Changelog

• Support for custom CA certificates (important: An application release is necessary to fully roll out this feature. You can ask your Gitpod Account Manager whether your instance has received the required release)
• Disabled scaling the instance to 0 nodes during working hours (6:00 to 22:00 local time to the instance) to speed up the workspace starts in the morning. Scale to 0 is still enabled on weekends.
• Improvements of log groups associated with Lambda functions to reduce cost and align function names with AWS conventions. This requires all lambdas to be recreated, leading to the longer than usual clean up time mentioned above.
• Enforce use of IMDSv2 AWS metadata endpoint for EC2 instances
• Various bug fixes

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following changes:

Changes to the stack for the role used to execute the Gitpod CF template:

Changes to Gitpod CF template

How to update

• Follow the process laid out in Updating the Enterprise Infrastructure

• This update does not impact running workspaces and can be done during working hours.

Changelog

• Update to the application controller (Lambda) to improve the ordering of its operations
• Turn off debug mode for the telemetry controller as it was logging too much
• Turn off AZ rebalancing which was impacting the stability of some nodes and thus workspaces
• Set workspace DNS resolvers to be local VPC resolver IP instead of public DNS lookup. This resolves networking issues in environments where public DNS lookups are blocked. This is the first of a two part roll out process, the second part is an application change.

Expected CloudFormation Change Set

The change set being generated as part of this CF change is expected to include the following 14 changes: