Gitpod Workspace Networking

Dec 16, 2021

Gitpod Workspace Networking

Avatar of csweichel Christian Weichel Avatar of aledbf Manuel Alejandro de Brito Fontes

Gitpod workspaces are Kubernetes pods. Each Kubernetes pod maintains its own network namespace - similar to how a regular container would. Network namespaces (netns) provide isolation of the system resources associated with networking.

Container runtimes (e.g. Docker) or VPNs (e.g. Tailscale) both require control over these system resources associated with networking. If we wanted to provide workspaces with control over the network namespace of their pod, that would require CAP_NET_ADMIN on the node - clearly not a good idea.

Enabling rootless Docker in workspaces

More than a year ago we enabled “$lib/components/workspacekit).

Docker-specific network setup

You’ll have noticed that a network namespace was not part of additions we make to what Kubernetes provides. Docker however, requires the control a network namespace would afford.

In a Gitpod workspace, we don’t just start a Docker daemon, but instead call docker-up. This custom piece of code produces a network namespace using slirp4netns, and starts the docker daemon in there. The combination of the user namespace we provide using workspace, and this newly created network namespace, enables docker to create the networking devices and iptables entries it requires.

Wait, what’s slirp4netns, and why do we need this?

Creating a network namespace is the easy part and does not require a lot of permissions (capabilities). You can easily try that for yourself using unshare -n bash. You’ll quickly find that it isolates a little too well, and that you don’t have any connection to the world outside of the namespace you’ve just created.

If you have enough capabilities/permissions outside of the network namespace, you can just create a veth pair that effectively tunnels traffic from within the netns to outside of the netns. However, we’re creating the netns because we don’t have those permissions in the first place.

Slirp4netns solves this problem using TAP/TUN, by turning itself into a user-space packet forwarder. Running outside the netns (hence it has access to the network of the “outside world”), it creates a tun device inside the network namespace. All packets to and from this tunnel network device are forwarded through user-space now.

Biggest benefit: works without elevated capabilities. Biggest drawback: forwards all traffic through user-space, which incurs a significant performance cost.

CAP_NET_ADMIN for everyone those who configure it

We’ve been living with the setup described above (netns only for the Docker daemon) for more than eight months. Mainly driven by performance considerations, but also because we saw no use-case which would warrant a change.

One day, we were approached by the awesome folks from Tailscale. They were working on how to make Tailscale available in Gitpod workspaces. The underlying issue: creating a TAP/TUN device and setting up routes to make use of it, all within a regular Gitpod workspace.

workspace-wide network namespace

We had solved this problem already for one specific use-case: the Docker daemon. Within a single day, we wrapped the entire workspace in a network namespace, if the workspace had a new experimentalNetwork: true config flag set.

Why make this optional behind a flag? Because this is a considerable change of the workspace networking setup, and we are careful not to break the existing experience. Also, the slirp4netns solution comes with a performance penalty (see above).

However, with this change, we no longer needed to special case Docker, and Tailscale works out of the box.

CAP_NET_ADMIN for everyone

After observing the behaviour of this new networking setup in the wild, and with our recent move to k3s, we felt comfortable enough to make this behaviour standard. On the 9th December, we rolled out a change which enabled this network namespace addition for all workspaces. We’ve also seen this change fixing other networking related issues, providing even more reasons to move ahead.

Today, when you start a workspace, sudo tailscaled will just run, and your Docker daemon will share the same network setup as the rest of the processes in your workspace. You can even do the networking equivalent of rm -rf / and break your workspace by running sudo ip link set tap0 down (you’ll have to stop your workspace from the dashboard, or let it time out).

There’s still the performance cost of slirp4netns we have to deal with. Now that we run on more recent Linux versions, we can integrate a seccomp-addfd approach and follow what bypass4netns implements.

Exciting times

It’s super exciting to see how quickly these changes have come together, and the use-cases they enable. We’re nowhere near the end of what we can do in this space, but are just getting started. If you want to join the fun, please head over to gitpod.io/careers and see what floats your boat.

Share this post:

  • Twitter
  • Reddit
  • HackerNews

More articles

DevX Digest 06 - Offload pressure to improve DevX

For our sixth edition of DevX Digest, we’d be breaking down the key highlights of our DevXPod conversation with Shanea from CodeSee.

Avatar of pawlean Avatar of mikenikles Jan 13, 2022

DevX Digest 05 - How DevX has evolved

For our fifth edition of DevX Digest, we’d be breaking down the key highlights of our DevXPod conversation with Jamon from Infinite Red.

Avatar of pawlean Avatar of mikenikles Dec 21, 2021

DevX Digest 04 - Building a Developer Experience team

For our fourth edition of DevX Digest, we’d like to introduce our new podcast with a first episode all about building a developer experience team!

Avatar of pawlean Avatar of mikenikles Avatar of loujaybee Dec 6, 2021