Why CDEs should be prioritized before IDPs

Lou Bichard Product Manager at Gitpod

Nov 12, 2024

Congratulations on becoming head or founding engineer of an early platform team! Whether you asked for this role or simply inherited it through working in infrastructure, I imagine you’re now facing pressure to deliver a roadmap. With your reputation on the line, you’re probably wondering “where do I start?” while considering investments in things like CI/CD or DORA.

In your search you may have come across internal developer portals (IDP) like Backstage and cloud development environments (CDEs) like Gitpod. At conferences I’m often asked about the difference. While both boost developer productivity, they represent fundamentally different approaches.

In short: IDPs focus on organizing your ecosystem top-down, while CDEs transform developer work from the ground-up. In this article we break down the differences between CDEs and IDPs but also critically we look at why CDEs should be a priority alongside, or even before your IDP.

It all started with internal developer portals

Let’s go back to 2019 when Spotify announced what is now a hugely popular platform tool: Backstage. I want to start our conversation specifically with Backstage as it’s probably the most talked about internal developer portal to date and is nearly always high on the “to investigate” list for many platform teams.

So, what is Backstage? In a very literal sense, it’s a Typescript front and back end application that ingests metadata about software in your company. This creates a “software catalog”. It could be information about your microservices, SDKs, libraries, etc. You suck all of that information and data up and you put it into an interface. This catalog then forms a sort of “backbone” for your platform. A structured foundation that you can use as a jumping off point to implement lots of other functionality such as documentation or showcasing build information.

Here’s how Backstage looks.

As you click into each entity you see more details and can dig further. You can extend the interfaces with additional views from marketplace plugins or write your own.

Here’s an example metadata file that you could use to power your catalog:

yml

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
    name: artist-web
    description: The place to be, for great artists
    labels:
        example.com/custom: custom_label_value
    annotations:
        example.com/service-discovery: artistweb
        circleci.com/project-slug: github/example-org/artist-website
    tags:
        - java
    links:
        - url: https://admin.example-org.com
          title: Admin Dashboard
          icon: dashboard
          type: admin-dashboard
spec:
    type: website
    lifecycle: production
    owner: artist-relations-team
    system: public-websites

But—let’s not get sidetracked. There are lots of ways to specify your catalog in Backstage.

The important thing is that you understand that Backstage works by adding metadata about your software which in turn drives the Backstage user interface. That user interface is then used by architects, developers, managers and the like to answer questions like: who owns this service? What data does this API expose? Who is on call for this component?

This flexibility is both Backstage’s superpower and its achilles heel. If you think Backstage will be a quick npm install and terraform apply to install and get running, you’re going to have a bad time. Backstage is complex. It requires knowledge of TypeScript (which most platform teams simply don’t have) and you’ll likely need to build your own plugins where you’ll need to host it and operate it—the list ultimately goes on.

The rise of out-of-the-box internal developer portals

Architecturally, Backstage is designed in a way that is better suited to very large engineering organizations more so than your run-of-the-mill developer organization. The graveyard of failed Backstage initiatives is growing due to platform teams that overestimate the sheer undertaking that is required to get it off the ground and supported in an organization.

Which is why we now see tools like Cortex and Port, where part of the main value is that they’re easier to set up and operate. This is particularly compelling for organizations that simply don’t have the headcount to dedicate to operating Backstage and building custom plugins.

However, with the growth of the market, what constitutes an internal developer portal is continuing to increase in scope. Internal developer portals are now pushing for developers to “take more action” inside the portals, and so we see more and more features like applying infrastructure, and triggering workflows. This makes internal developer portals feel more like automation platforms like GitLab, rather than their original roots as catalog software.

It’s this push towards integration of automation that seems to now be blurring the lines between internal developer portals and tools like cloud development environments. However, these tools come at the challenges of developer productivity from very different angles.

What is a cloud development environment?

A cloud development environment allows you to standardize and automate everything related to the setup and management of your development environments. Depending on how many meetings your developers have, they are likely spending 30+ hours writing code, running tests and going through this in iterative loops.

With a CDE, developers and platform teams can come together to automate away the toil in setting up environments to ultimately reduce getting started time to zero. In real terms, with a CDE like Gitpod, you define all the steps required to start any project, from your dependencies to access credentials. You store these declarative files either directly next to your code in your repository, or in a centralized platform repository that can be accessed to apply across your organization. This enables platform team’s and developers to automate development workflows like:

• Dependency managements
• Access management
• Resource configuration
• Security and compliance controls (that can be applied globally)

Here’s how Gitpod looks:

All of which lead to the ultimate goal of improving developer productivity while simultaneously improving security—not compromising it.

And here’s a very simple example of the configuration used to power Gitpod. These definitions can be stored alongside the source code in a repository, or centralized if needed to be owned by a platform team.

yml

services:
    database:
        name: PostgreSQL
        commands:
            start: docker run postgres
tasks:
    run-unit-tests:
        name: Runs unit tests
        command: go test -v ./...

See Automations for more information.

Similarities between CDEs and IDPs

There are therefore clear similarities between the goals of an internal developer portal and a cloud development environment:

• Developer onboarding: Both tools improve developer onboarding time. Cloud development environments automate the time to set up your environment and connect the wider platform tools. Internal developer portals help developers understand the broader ecosystem such as other services, APIs and teams.
• Developer self-service: Both tools promote the idea of developer self-sufficiency and self-service to avoid bottlenecking on a central team. With a CDE developers can start their environments without raising a ticket and platform teams can help developers to automate their setup and integrate with the development ecosystem.
• Facilitating standardization: Internal developer portals help to standardize more at the ecosystem level on things like pull request process and ensuring deployment steps are adhered to whereas cloud development environments standardize at the environment level in terms of what access credentials developers use, what dependencies of software and packages they are using, etc.

The critical difference between a developer portal and a cloud development environment is where they start when looking to improve developer productivity: internal developer portals look to the internal ecosystem while cloud development environments look to developer workflows directly. Cloud development environments operate on where developers spend the majority of their time each day, whereas internal developer portals look to improve the ecosystem around developers—primarily affecting managers, architects, VPs and others.

Aside: There are significant security reasons for companies to adopt CDEs such as reducing their attack surface area for threats like source code exfiltration, and getting greater control over who in your organization has access to what source code, which secrets, which networks, etc.

Optimizing your development environments should be a priority that comes alongside—or even before your IDP

Starting with an Internal Developer Platform (IDP) without optimizing your development environments at the same time can be counterproductive in the long-run for your platform engineering and developer productivity efforts. As you may already know, developer productivity is best measured by: time spent in flow, reduced cognitive load and fast feedback loops.

Our goal as platform engineers should be maximizing the time our developers spend in their editors and we should aggressively work to minimize time spent on administrative overhead and yak-shaving. While IDPs are often a necessary evil to support standardization and management decision-making they require developers to context-switch and constantly break their flow. If we’re not careful we might introduce more—and not less friction.

The temptation with an IDP is to push for portal logins as a proxy for success. But at some point developers context-switching from their editor to other tools including their IDP needs to be actively minimized—not expanded. Without a CDE in place, the urge to build more and more functionality into the IDP will be very strong, but if we’re not careful we actively undermine our global goals of driving developer productivity by over-indexing on features in our IDP that break developers workflows. Our IDP should therefore adhere to the principle of “thinnest viable platform”.

The path of least resistance for expanding our platform capabilities should be directly integrating with our development environments—not tools outside developer workflows. For example, when running tests developers shouldn’t need to leave their editor to find an execution button in a different interface and watch logs through an interface only to swap back to their development environment to replicate this setup to continue their task. This type of automation work should be done directly from the development environment.

All of this is not to say that internal developer portals are not useful—they very much are. Because IDPs optimize the ecosystem around the developer. Yet to unlock those significant productivity gains we’re after we must also invest in our development environments either simultaneously, or ahead of our IDP. Because even with a flawless IDP implementation we are still accidentally ignoring the biggest opportunity for developer productivity, which is where our developers spend the majority of their time: their development environment.

If you’re interested to learn about how CDEs could impact your organization see our whitepaper on: The ROI of CDEs.