Enhancing Deployment Safety at GitHub with eBPF: Breaking Circular Dependencies

Introduction: The Circular Dependency Dilemma

At GitHub, we host all of our own source code on github.com—a practice that makes us our own biggest customer. By testing changes internally before rolling them out to users, we ensure a robust experience. However, this approach introduces a notable risk: if github.com experiences downtime, we lose access to the very code we need to fix it. This creates a simple circular dependency: to deploy GitHub, we require GitHub itself. We mitigate this by maintaining a mirror repository for fixing forward and pre-built assets for rollback scenarios.

But the challenges don't end there. Even with a mirror, deployment scripts can inadvertently introduce new circular dependencies. For instance, a script might depend on an internal service that itself relies on GitHub, or it might attempt to download a binary from GitHub during an outage. To address these issues, we designed a new host-based deployment system that leverages eBPF (extended Berkeley Packet Filter) to monitor and selectively block problematic calls. This blog post explores our findings and provides a practical guide to writing your own eBPF programs for deployment safety.

Types of Circular Dependencies

Consider a hypothetical scenario: a MySQL outage prevents GitHub from serving release data from repositories. To resolve the incident, we need to roll out a configuration change to the affected stateful MySQL nodes by executing a deploy script on each node. During this process, several types of circular dependencies can arise:

Direct Dependency

The MySQL deploy script attempts to pull the latest release of an open source tool from GitHub. Since GitHub cannot serve the release data due to the outage, the script fails to complete. This is a straightforward dependency—the deploy tool depends on the very service that is down.

Hidden Dependency

The MySQL deploy script uses a servicing tool already present on the machine's disk. However, when the tool runs, it checks GitHub to see if an update is available. If it cannot contact GitHub due to the outage, the script may fail or hang, depending on how the tool handles the update-check error. This dependency is not explicit in the script but emerges from the tool's internal behavior.

Transient Dependency

The MySQL deploy script calls, via an API, another internal service (for example, a migrations service). That service, in turn, attempts to fetch the latest release of an open source tool from GitHub to use the new binary. The failure propagates back to the deploy script, causing it to stall or crash.

How eBPF Solves These Dilemmas

Previously, the responsibility fell on each team owning stateful hosts to manually review deployment scripts and identify circular dependencies. In practice, many dependencies remain undetected, leading to deployment failures that could be avoided. With eBPF, we can apply a systematic, runtime-based approach.

eBPF allows us to insert small, safe programs into the Linux kernel to observe and control system behavior. For deployment safety, we use eBPF to monitor network calls made by deployment scripts and selectively block those that would create circular dependencies. For example, we can block any outbound connection from a deploy script to github.com (or to internal services that depend on github.com) during an incident.

Here’s how we implemented it at GitHub:

• Monitoring: We attach eBPF programs to system calls like connect and sendto to inspect destination IPs and ports.
• Filtering: Based on a pre-defined list of forbidden endpoints (e.g., github.com, certain internal APIs), we either allow or deny the call.
• Logging and Alerting: All blocked calls are logged for post-incident analysis, helping teams identify hidden dependencies.

This approach does not require modifying the deployment scripts themselves, making it easy to adopt across teams. It also works transparently—developers write normal scripts, and eBPF enforces the safety policies in the kernel.

Getting Started with eBPF for Deployment Safety

You can start writing your own eBPF programs to prevent circular dependencies. Here are the key steps:

1. Set up the eBPF toolchain: Install bcc (BPF Compiler Collection) or use libbpf with CO-RE (Compile Once – Run Everywhere).
2. Write a simple program: Create a BPF program that hooks into the connect syscall. For example, in Python with bcc:

from bcc import BPF
bpf_text = """
int kprobe__sys_connect(struct pt_regs *ctx, int sockfd, struct sockaddr *addr, int addrlen) {
    // Parse IP and decide to block
    return 0;
}
"""
b = BPF(text=bpf_text)
# ... attach and read logs

3. Define forbidden endpoints: Maintain a list of IP addresses or domain names that are considered dangerous during incidents.
4. Test and deploy: Run the eBPF program on a staging environment first. Use it to block calls to the forbidden endpoints and verify that deployed scripts fail safely (e.g., with a clear error message) rather than hanging.

For a deeper dive, check out our open-source tools and documentation linked below.

Conclusion

Circular dependencies are a subtle but critical threat to deployment reliability, especially when your infrastructure runs on its own platform. By leveraging eBPF, GitHub has moved from manual script reviews to automatic, kernel-level enforcement of deployment safety. This not only reduces incident response time but also catches hidden and transient dependencies that might otherwise go unnoticed. We encourage you to experiment with eBPF for your own deployment pipelines—it’s a powerful tool for building more resilient systems.