GitHub has announced a new Git storage architecture, codenamed “Project Rubicon,” designed to support thousands of writes per second on a single repository, as the platform prepares for a future where AI agents drive sustained parallel development. The architecture, described in a blog post by Karthik Raman, has entered limited private preview and is slated for broader rollout early next year.
The redesign was built to address demands from agentic development. According to GitHub, the busiest repositories currently receive at most dozens of writes per second in short bursts, but the company expects that rate to become sustained within a year as agents work in parallel around the clock. GitHub’s legacy architecture had tightly coupled components and a consensus group. Virtual replicas were the first step toward the new architecture and a means of onboarding repositories.
The new architecture centres on two principles: minimise coordination and decouple storage from compute. Git objects are stored as packs in Azure Blob Storage, while references and the pack manifest live in a separate replicated log service called Ref-CAS, which enables a small consensus group to order reference updates independently.
Under the new design, writes are handled by virtual replicas, disposable compute nodes that index incoming packs, upload them to Blob Storage, and then commit a reference transaction to Ref-CAS. Reads check a version number; an unchanged repository costs a single comparison. Compaction and garbage collection run in the background on separate nodes, directly against durable storage, so that serving nodes can dedicate resources to requests.
GitHub said the architecture has demonstrated a flat latency curve as writes climb, with compaction scaling horizontally across shards to keep the pack count effectively flat. The company also outlined a roadmap of further improvements, including fetching single objects from Blob Storage, organising packs by object type, and placing replicas in any region with bounded staleness.
Analysis
Why This Matters
- Agents writing code in parallel can push to the same repository at rates that overwhelm current Git infrastructure; Rubicon aims to keep pace without compromising latency or reliability.
- The architecture decouples storage from compute, allowing each part to scale independently, which could make large monorepos more viable for teams that adopt agent-driven workflows.
- GitHub’s commitment to one architecture for all repositories, from side projects to monorepos, means improvements reach every user automatically.
Background
The rise of AI-powered coding agents has changed what a busy repository looks like on GitHub. Agents generate code, submit pull requests, and trigger CI builds around the clock, producing sustained write rates that far exceed the bursty patterns of human developers. GitHub’s legacy architecture, which co-located storage, request serving and repository maintenance on tightly coupled replicas, could not scale to thousands of writes per second on a single repository without fundamental redesign.
What to Watch
- The limited private preview: how initial users report latency and reliability under load.
- The broader rollout early next year: whether the architecture meets expectations for repositories of all sizes.
- Upcoming optimisations such as single-object fetches and global replicas, which could further reduce operational costs for agent workloads.
Checked
10 claims in this story checked against 1 source: 8 supported.
Show each claim
- SupportedThe headline names the architecture “Project Rubicon” and says it targets agent-scale performance.The GitHub Blog: “We called it “Rubicon,” a nod to the point of no return the industry crossed with agentic development.”
- SupportedThe redesign is built to handle thousands of writes per second on a repository.The GitHub Blog: “For a single repository, that meant the new architecture needed to: Accept thousands of writes per second.”
- SupportedAI-agent activity is driving growth in repository traffic.The GitHub Blog: “As more of that work shifts to agents running in parallel and around the clock, we expect those bursts to become a sustained rate within the next year. Each push leads to more reads”
- Not in the sourcesThe blog post describing the architecture was written by Karthik Raman.The GitHub Blog
- SupportedThe architecture is in limited private preview, with broader rollout early next year.The GitHub Blog: “It’s available in a limited private preview now, with a broader rollout early next year.”
- SupportedThe busiest repositories currently receive at most dozens of writes per second, typically in short bursts, and GitHub expects those bursts to become sustained within the next year.The GitHub Blog: “Today, the busiest repositories on GitHub.com receive at most dozens of writes in a single second, typically in short bursts. As more of that work shifts to agents running in parallel and around the clock, we expect those bursts to become a sustained rate within the next year.”
- SupportedVirtual replicas handle writes by indexing incoming packs, uploading them to Blob Storage, and committing a reference transaction to Ref-CAS.The GitHub Blog: “a virtual replica indexes the incoming pack and uploads it to Blob Storage, then commits one transaction to Ref-CAS that lists the pack in the manifest and updates the references.”
- Partly supportedCompaction and garbage collection both run on separate nodes directly against durable storage.The GitHub Blog: “Separately, a fleet of compaction nodes works in the background, directly against Blob Storage and Ref-CAS”
- SupportedThe company demonstrated a flat latency curve as writes increase, while compaction keeps pack count effectively flat as pushes rise.The GitHub Blog: “Using these approaches, we’ve been able to consistently demonstrate a flat latency curve as writes climb”
- SupportedPlanned improvements include fetching single objects from Blob Storage, organising packs by type, and placing replicas in any region with bounded staleness.The GitHub Blog: “Fetch a single object from Blob Storage instead of the whole pack that holds it. Organize packs by type. Group commits, trees, and blobs into their own packs.”
