​
Introduction

Managing multiple Kubernetes clusters across different customer environments presents significant operational challenges. The Enhanced Cluster Management feature was introduced to streamline this process, enabling Gitpod to provide superior support while maintaining strict security boundaries and full transparency.

​
Key Benefits

  • Reduced resolution time for technical incidents
  • Full visibility into cluster operations and security boundaries
  • Decreased operational overhead for your organization

​
Opting Out

While we recommend keeping Enhanced Cluster Management enabled, you can choose to disable this feature. Before doing so, please note:
  • Incident resolution may take longer
  • Your organization may be required to do manual intervention
  • Support processes may require additional steps
To disable the feature:
  1. Access CloudFormation in your installation’s AWS account and region
  2. Select your “Instance” stack and “Update” it
  3. Set the “Opt-out of enhanced cluster management” parameter to “Yes”
Questions about disabling this feature? Your Account Manager is ready to help evaluate the best approach for your organization.

​
How It Works

The Enhanced Cluster Management feature operates through a secure, asynchronous mechanism:
  • A management agent runs within your Kubernetes cluster
  • The agent monitors a cell-specific S3 bucket for command requests
  • When a command is detected, the agent:
    • Executes the approved command in the cluster
    • Sanitizes the output to remove sensitive information
    • Writes the sanitized response back to the S3 bucket
  • All activities are logged for full auditability

​
Security and Data Protection

  • Your cluster remains private with no direct external access
  • All communication occurs through a S3 bucket owned by Gitpod and its Gitpod Enterprise control plane. Read access to this bucket will be limited to the AWS Account of the customer’s Gitpod installation.
  • Response data is sanitized at the source before being written to S3
  • Commands are limited to a pre-approved set defined by RBAC policy defined and reviewable in the Infrastructure Template
  • You maintain full audit capability through S3 logs

​
Allowed Operations

The following operations are permitted based on the RBAC policy:

​
Read Operations

  • List and view: pods, services, nodes, jobs, namespaces, persistent volumes, events
  • Access pod logs
  • View deployments, daemonsets, replicasets, statefulsets
  • Monitor metrics and custom metrics

​
Management Operations

  • Delete pods when necessary
  • Update node status
  • Scale deployments and replicasets
  • Create pod evictions for maintenance
  • Create jobs for maintenance

​
Workspace Management

  • View and manage Gitpod workspaces
Please verify the full policy in the Cloudformation Template

​
Auditing Activity

To review commands executed you can access audit logs via AWS CloudWatch. There are two EKS clusters, meta and workspace, logs for Enhanced Cluster Management are stored in log groups /aws/containerinsights/meta/audit and /aws/containerinsights/workspace/audit. Logs contain detailed information about requests and responses.

​
Accessing Audit Logs with AWS Logs Insights

Follow these steps to view audit logs in AWS CloudWatch using Logs Insights:
  1. Open the AWS Management Console and navigate to CloudWatch.
  2. In the left navigation pane, select Logs Insights.
  3. Choose the log group /aws/containerinsights/meta/audit and/or /aws/containerinsights/workspace/audit.
  4. Use the following query to filter relevant logs: 
    fields @timestamp, log_processed.request.cmd, log_processed.request.args.0, log_processed.request.args.1, log_processed.response.output
| filter kubernetes.container_name='genie'
| sort @timestamp desc
| limit 10000
  5. Click Run Query to retrieve and analyze logs.

​
Example Query Output

When executing the query, the results may appear as follows:
FieldValue
……
@timestamp1739363063721
……
kubernetes.container_namegenie
log_processed.request.cmdkubectl
log_processed.request.args.0get
log_processed.request.args.1pods
log_processed.request.args.2-n
log_processed.request.args.3default
log_processed.response.output
NAME READY STATUS RESTARTS AGE
blobserve-5566c59d86-kc5tp 2/2 Running 0 5d2h
blobserve-5566c59d86-mw2dn 2/2 Running 0 5d2h
content-service-79b86d5779-gvkb7 2/2 Running 0 5d2h
content-service-79b86d5779-qv25t 2/2 Running 0 5d2h
dashboard-6f7576dcf6-7hvpx 1/1 Running 0 2d1h
dashboard-6f7576dcf6-msd9s 1/1 Running 0 2d1h
ide-metrics-7f4bc56c5f-ntwll 2/2 Running 0 5d2h
ide-proxy-6b465b4bb-blqvw 1/1 Running 0 5d2h
ide-proxy-6b465b4bb-g8zt4 1/1 Running 0 5d2h
ide-service-68cb94b69c-kb9nx 2/2 Running 0 5d2h
ide-service-68cb94b69c-kn7lx 2/2 Running 0 5d2h
openvsx-proxy-0 3/3 Running 0 5d2h
openvsx-proxy-1 3/3 Running 0 5d2h
proxy-8467bbd84-k7zjf 2/2 Running 0 6d20h
proxy-8467bbd84-nqqq2 2/2 Running 0 6d20h
public-api-server-58b4596874-7cctc 2/2 Running 0 5d2h
public-api-server-58b4596874-znfcg 2/2 Running 0 5d2h
redis-66d4f97ccc-w7slh 3/3 Running 0 6d20h
server-5cdf984cbc-9qgjw 2/2 Running 0 25h
server-5cdf984cbc-ptkv7 2/2 Running 0 25h
spicedb-578b64855f-fr252 2/2 Running 0 5d2h
spicedb-578b64855f-tldmb 2/2 Running 0 5d2h
usage-59c8996f78-nbghd 2/2 Running 0 5d2h
usage-59c8996f78-p8th5 2/2 Running 0 5d2h
ws-manager-bridge-8459d6d995-8lxxz 2/2 Running 0 5d2h
This allows you to track cluster activities and ensure accountability for executed commands.