Enhanced cluster management

​

Introduction

Managing multiple Kubernetes clusters across different customer environments presents significant operational challenges. The Enhanced Cluster Management feature was introduced to streamline this process, enabling Gitpod to provide superior support while maintaining strict security boundaries and full transparency.

​

Key Benefits

• Reduced resolution time for technical incidents
• Full visibility into cluster operations and security boundaries
• Decreased operational overhead for your organization

​

Opting Out

While we recommend keeping Enhanced Cluster Management enabled, you can choose to disable this feature. Before doing so, please note:

• Incident resolution may take longer
• Your organization may be required to do manual intervention
• Support processes may require additional steps

To disable the feature:

1. Access CloudFormation in your installation’s AWS account and region
2. Select your “Instance” stack and “Update” it
3. Set the “Opt-out of enhanced cluster management” parameter to “Yes”

Questions about disabling this feature? Your Account Manager is ready to help evaluate the best approach for your organization.

​

How It Works

The Enhanced Cluster Management feature operates through a secure, asynchronous mechanism:

• A management agent runs within your Kubernetes cluster
• The agent monitors a cell-specific S3 bucket for command requests
• When a command is detected, the agent:
  • Executes the approved command in the cluster
  • Sanitizes the output to remove sensitive information
  • Writes the sanitized response back to the S3 bucket
• All activities are logged for full auditability

​

Security and Data Protection

• Your cluster remains private with no direct external access
• All communication occurs through a S3 bucket owned by Gitpod and its Gitpod Enterprise control plane. Read access to this bucket will be limited to the AWS Account of the customer’s Gitpod installation.
• Response data is sanitized at the source before being written to S3
• Commands are limited to a pre-approved set defined by RBAC policy defined and reviewable in the Infrastructure Template
• You maintain full audit capability through S3 logs

​

Allowed Operations

The following operations are permitted based on the RBAC policy:

​

Read Operations

• List and view: pods, services, nodes, jobs, namespaces, persistent volumes, events
• Access pod logs
• View deployments, daemonsets, replicasets, statefulsets
• Monitor metrics and custom metrics

​

Management Operations

• Delete pods when necessary
• Update node status
• Scale deployments and replicasets
• Create pod evictions for maintenance
• Create jobs for maintenance

​

Workspace Management

• View and manage Gitpod workspaces

Please verify the full policy in the Cloudformation Template

​

Auditing Activity

To review commands executed in your cluster, you can access audit logs via AWS CloudWatch. These logs are stored in the log group /aws/containerinsights/meta/audit and contain detailed information about requests and responses.

​

Accessing Audit Logs with AWS Logs Insights

Follow these steps to view audit logs in AWS CloudWatch using Logs Insights:

1. Open the AWS Management Console and navigate to CloudWatch.

2. In the left navigation pane, select Logs Insights.

3. Choose the log group /aws/containerinsights/meta/audit.

4. Use the following query to filter relevant logs:

   fields @timestamp, log_processed.request.cmd, log_processed.request.args.0, log_processed.request.args.1, log_processed.response.output
   | filter kubernetes.container_name='genie'
   | sort @timestamp desc
   | limit 10000
   

5. Click Run Query to retrieve and analyze logs.

​

Example Query Output

When executing the query, the results may appear as follows:

NAME READY STATUS RESTARTS AGE
blobserve-5566c59d86-kc5tp 2/2 Running 0 5d2h
blobserve-5566c59d86-mw2dn 2/2 Running 0 5d2h
content-service-79b86d5779-gvkb7 2/2 Running 0 5d2h
content-service-79b86d5779-qv25t 2/2 Running 0 5d2h
dashboard-6f7576dcf6-7hvpx 1/1 Running 0 2d1h
dashboard-6f7576dcf6-msd9s 1/1 Running 0 2d1h
ide-metrics-7f4bc56c5f-ntwll 2/2 Running 0 5d2h
ide-proxy-6b465b4bb-blqvw 1/1 Running 0 5d2h
ide-proxy-6b465b4bb-g8zt4 1/1 Running 0 5d2h
ide-service-68cb94b69c-kb9nx 2/2 Running 0 5d2h
ide-service-68cb94b69c-kn7lx 2/2 Running 0 5d2h
openvsx-proxy-0 3/3 Running 0 5d2h
openvsx-proxy-1 3/3 Running 0 5d2h
proxy-8467bbd84-k7zjf 2/2 Running 0 6d20h
proxy-8467bbd84-nqqq2 2/2 Running 0 6d20h
public-api-server-58b4596874-7cctc 2/2 Running 0 5d2h
public-api-server-58b4596874-znfcg 2/2 Running 0 5d2h
redis-66d4f97ccc-w7slh 3/3 Running 0 6d20h
server-5cdf984cbc-9qgjw 2/2 Running 0 25h
server-5cdf984cbc-ptkv7 2/2 Running 0 25h
spicedb-578b64855f-fr252 2/2 Running 0 5d2h
spicedb-578b64855f-tldmb 2/2 Running 0 5d2h
usage-59c8996f78-nbghd 2/2 Running 0 5d2h
usage-59c8996f78-p8th5 2/2 Running 0 5d2h
ws-manager-bridge-8459d6d995-8lxxz 2/2 Running 0 5d2h

This allows you to track cluster activities and ensure accountability for executed commands.