Picture this: It’s mid-2026. You own developer velocity at an economy-of-scale startup. Software engineers are the fixed cost around these parts, so the company gives them API keys for fancy coding agents. They are tasked with automating the processes that support the company’s geometric growth towards the minimum efficient scale required to color the balance sheet bright green. They need to ship yesterday.

Your colleagues learn the trick pretty fast: Parallel coding agents pushing, failing, fixing, and delivering stacks of pull requests (PRs) with passing continuous integration (CI) checks. The most combative submit north of 200 PRs a week.

Getting there is the easy part. If you sat through Christopher Nolan’s The Odyssey (with apologies for the 3,000-year-old spoilers if you haven’t), you know the rest of the story: the first act wraps up a ten-year war, and the getting-home takes the other two and a half hours. This is what happens to organizations that switch to agents: CI-passing PRs written at record speed, then hours or days getting reviewed and merged to production.

This was us at Base a month ago, so we rebuilt that path. This post is about how.

Step 1: Push your post-submit left

Let’s start with the obvious: no human comprehensively reviews 200 PRs a week. We use an array of AI review agents, which helps. Importantly, we also decided to shift as much of the review burden as possible into tests: we leaned into a monorepo and now aggressively test every atomic change before it merges, when feasible. More specifically, we made the uncommon choice to pull end-to-end suites that traditionally run post-merge, nightly, or before a release into the pre-merge CI gate. This way, a change gets in by surviving a well-designed test workload, not by being well-vetted.

To be clear, we’d have argued against this before agents, but felt the calculus had flipped. Anyways, does this help?

Good news: Provided design discussions happen somewhere other than pull requests and the tests actually match your risk/reliability posture, this helps. Minor caveat: agents write the tests too, so a change could weaken its own gate. We trust reviewers to catch it if one does; test changes still require human attention. You’ll generally need a somewhat sophisticated set of change approval rules to protect your most precious lines of code.

Bad news: CI is now even slower! A PR or merge validation run can have a long wall-clock time, require an unreasonable number of machine-hours, or be inherently flaky (e.g., for tests that run on unreliable hardware). That cost is the villain of the rest of this post. Giving up the testing was off the table, because the testing is the only reason any of this is safe; the cost had to come down another way.

Step 2: Only run the tests you need, and save the footprint

This part isn’t exactly brain surgery: Don’t run the whole suite on every change, only the part the change can actually affect. At Base, we use an industry standard input- and content-addressed hermetic build system: Google’s Bazel, paired with Tinder’s bazel-diff. See this presentation if this is unfamiliar.

Besides making pre-submit CI faster and cheaper, this produces a footprint of the PR, which turns out to matter: for a given change, a machine-computed answer to “what does this change touch relative to the reference point chosen by the CI run?”. Hold that thought, but for now: does this work?

Good news: Steps 1 and 2 fixed the left half of that first timeline. PRs now go green as fast as their affected tests go, cheaply, and with less of a human bottleneck.

Bad news: All that did for the right half is make it worse! Approved PRs now arrive by the hundreds, and every one of them still has to make its way to the same trunk branch. The bottleneck shifted right yet again, into the merge strategy.

Step 3: Hear the sirens’ song

This step isn’t really actionable; it’s more of an explainer to motivate what follows, in case you’re not clear about what merge queues are. Recall that a green checkmark is a statement about the past. Your PR was validated against the trunk branch of an hour ago, and thirty other PRs landed on the trunk branch since. Merging now means publishing a combination of changes that was never tested. Physical conflicts are caught by Git, but logical conflicts (two changes that each pass independently but break together) could slip through.

So your trunk branch needs a guarantee: no combination of changes gets published without being validated first. There are plenty of fine implementations for serializing this validation, including GitHub's Merge Queue and GitLab's Merge Train. Unfortunately, this is slow at agent volume, even with diff-aware selective testing and caching:

• Once a PR triggers a heavy integration test suite, every PR following it pays the wall-time.

• Logical conflicts and flaky failures cause queue rejections; every PR that follows them has to restart its validation run.

Note that this problem is mostly a consequence of step 1, which parked the heavy suites in front of the merge. This means that every engineer stuck in that line is hearing voices: Nothing that landed since has anything to do with your change. It’s fine. Just merge it..

No matter how appealing, better plug your ears and row, because every time a logical conflict lands, it blocks CI and treats the entire team to a front-row seat at the revert. So how do we solve it?

Step 4: Enter parallel merge queues

The line in step 3 is serial because the queue assumes every merge can affect every PR behind it. Step 2 says otherwise: each change carries a footprint. If two footprints are disjoint, neither change can invalidate the other’s green, so the PRs can commute. Give each its own lane and validate both concurrently - only changes whose footprints intersect need serialization. The queue tracks this with a ledger: the footprint of every merge it lands on the trunk branch. Intersect a PR’s footprint with the merges since its base, and you know exactly whose green went stale.

At Base, we evaluated trunk.io and Mergify, which both offer parallel merge queues with dynamically configured lanes. Uber’s SubmitQueue also caught our attention with its aggressively speculative approach, for which an open-source rewrite is in progress.

Any of those will fix it! The long run stops being everybody’s problem: B merges as soon as its own fully cached validation clears, and if A is rejected, it takes down its own candidate, not the whole line.

That being said.. B ran nothing and waited for a full-cache-hit CI run anyways. Why queue it at all?1

Step 5: Directly merge to your trunk branch

Look at B’s row in the figure above. The intersection column already answered the only question that matters: nothing that landed since B’s base touches B’s footprint, so B’s green never went stale. And that answer cost one set intersection against the ledger. B’s lane (candidate branch, CI dispatch, a validation run made entirely of cache hits) re-derived a fact the queue already knew. When the intersection is empty, the safe move and the fast move are the same move: merge directly to main, no lane, no CI run. That optimization pushes us one notch further along the design space, without giving up correctness guarantees.

The decision rule is exactly the step 4 panel plus one column. For each green PR, intersect its footprint with everything the trunk branch has landed since its base and with everything currently validating in flight. Empty on both counts: merge directly, in seconds. Overlap with a landed merge: queue for revalidation in a lane, as before. A footprint that covers everything (a toolchain bump, say) serializes the world; that is the correct behavior, and it is rare2. Depending on your software architecture, a large majority of your PRs may merge directly to main.

We didn’t set out to build this; nobody wants to own a merge queue. But nobody sold the fast path either: a queue that uses the attested footprint to answer “must we test again?” and skips validation when the answer is no. Our current situation (about 100 people, agents everywhere, heavy integration tests) wanted it, so we built it3. It took an engineer about three weeks: two weeks from start to rollout, and another week of iterating on it in production towards stability and performance.

What we built is deliberately small: one Go service, about 16,000 lines plus a React dashboard and Postgres for state. A validation run is ordinary CI: push a candidate branch, dispatch the same workflow PRs run, squash-merge on green, record the footprint in the ledger. Overlapping PRs ride merge trains (the queued PR tree in the dashboard above), so even the slow path extracts parallelism.

A merge queue is concurrent, stateful, and guards your trunk branch, so we tested it like infrastructure. The whole system (real executor, real Postgres, fake GitHub) runs in-process, so every bug becomes a reproducible test case: overlap chains, merge trains, red builds, pushes mid-validation, footprint drift, external merges, and crashes injected at arbitrary points. Alongside those sits an exhaustively enumerated corpus of about 80,000 deterministic cases sweeping entry counts, footprint assignments, and train depths. On top of that sits model-based fuzzing: random multi-actor traffic (authors, CI, operators, a chaos actor) with global properties asserted after every command. The timing mattered, too: this work coincided with the release of Fable-class models4, which we exclusively used. In our estimate, the ROI of an internal build would not have been favorable before this model tier.

This overall approach resulted in an order of magnitude drop of our median time-to-trunk, which supported our contribution volume more than doubling since March. 40% of our merges skip validation entirely, and in our estimates, build graph improvements (tighter footprints, fewer over-broad targets) will take that north of 70%.

About the Author

Valentin Reis works on software infrastructure at Base Power. Previously, he worked on HPC systems at Argonne National Laboratory and helped build the software stack for Groq’s inference accelerator.

We just raised a Series D and are hiring for software roles. If you are interested in building fundamentals-driven solutions for a high growth vertical, click below.

See Careers

Becoming “America’s Next-Generation Power Company” is no small ambition — it’s a bet on ourselves, our team, and the future of energy in this country. Every team at Base Power pours their heart into innovations driving this mission, taking us from CAD files to batteries backing up homes across Texas. Deployments actualizes these efforts, turning grit into grid-connected hardware. Traditionally, battery, generator, and solar installs are treated like custom construction jobs. That doesn’t scale. At Base, we’re deadset on building hardware deployment like a factory: repeatable, efficient, and engineered for speed.

Challenges to Scale

When I joined, Base had deployed roughly 70 batteries. The first 50 installations were managed directly by our team on site. Even at 5 to 15 installs per week, due to numerous manual checkpoints and constant troubleshooting, the process was breaking and couldn’t keep pace with our ambitions. Our goal was bold: scale to 30+ installs per day with next-to-no direct contact or support, what we call “Base-Free” installs. To get there, we had to rethink processes, build tools, and team up across the company, all while holding fast to our high standards. The stakes were high: reliable, affordable electricity depends on nailing deployment at scale.

Key Blockers:

• Permitting and Regulatory: Getting permits for one address is hard. Securing permits daily for dozens of addresses across multiple jurisdictions that all enforce different code is a critical obstacle.

• Installer Capacity and Ability: Electricians are in short supply. How do we recruit, train, and oversee enough crews to meet demands while keeping quality and customer service top-notch?

• Supply Chain: We can’t run out of materials, but overstocking is not an option either. Once materials land, we’ve got to kit and deliver them statewide, every day.

• Commissioning, Day-of Issues, and Install Intervention: Systems need perfect testing to work, but customer tweaks, installer slip-ups, and troubleshooting means 3+ interventions per install.

• Lack of Tooling Causing Redundancy: Manual settings, updates, and tracking create slow, repetitive tasks.

• Hardware Issues: Equipment failures on or after install day can lead to difficult headaches down the road.

• Data Visibility: We needed clear, accurate data that everyone could see and use on the spot.

Rebuilding the Deployments Model for Scale

Making small tweaks to the system wasn’t an option. Here’s how we rebuilt the system from the ground up:

Full-Court Press

• Easing Regulatory Burden: We built a tool to automatically generate “permit packets” based on the battery configuration being built, equipping us for 100+ permits per day. We’re now working with policymakers to implement a system that standardizes code enforcement across the state.

• Adding Capacity while Maintaining Quality: Pooling directly from our installer network helped us quickly develop a reliable pipeline. We hired foundational master electricians to establish Base installer hubs in our large metro areas. We revamped manuals and built processes to hire, onboard, and prepare installers for field readiness.

• Acquiring a Warehouse, Onboarding an ERP: We set up a central warehouse in Austin and built its logistics to operate as the locus of Base operations. Next, we mapped delivery routes across Texas, ensuring our own trucks, warehouse personnel, and drivers make just-in-time deliveries to our members’ homes. An ERP system keeps our inventory dialed in as we grow, preparing Base for supply at scale.

• Empowering Installers with Tech: Without automation, manual system testing limits each installer to about six installs per day. Scaling beyond that would require hiring at the same pace as installation growth. Instead of hiring to match growth, we built a powerful, installer-facing app. It gives installers everything they need to bring a system online–commissioning, uploading close-out photos, accessing manuals, tracking progress, taking notes, and even troubleshooting independently. Several Base-free installs now happen every day.

• Establishing an Ecosystem: Alongside all of these changes, we developed an entire suite of helpful internal products. We informed the design of tools that significantly increased our ability to conduct site surveys, adjust battery settings, send address-specific firmware changes, track system issues fleetwide, and deploy batteries without intervention.

• Fixing Hardware Hiccups: By noting recurring issues and recovering failed hardware, we diagnosed and prevented system-critical failures such as water ingress through RJ45 connectors in battery modules or shorted PCBAs in hubs.

• See More: Visibility is everything when you’re coordinating dozens of installs and field services daily. We designed real-time dashboards for a clear view of installations and services, now widely used across teams.

These improvements were a relay across operations, software, hardware, fleet, and customer support. Every group brought their A-game, resulting in 10x-ing our daily installs while tracking for 50x by the end of the year. Deployments fuels Base’s push for energy abundance, scaling backup power to thousands of homes and boosting storage capacity at record pace, one household at a time.

About the Author

David “DWilly” Willson works on the Deployments team as a Deployments Engineer. He focuses largely on install operations and tooling improvements, occasionally moonlighting as a growth guy. He’s also referred to as Base’s “Chief Vibes Officer”.

The Fundamental Challenge

Retail electricity providers (REPs) operate in a world of structural risk asymmetry. We sell fixed-price contracts to customers while purchasing power from a market where prices can spike from $20/MWh to $5,000/MWh in minutes. This creates a potential for catastrophic loss that must be managed with precision.

The challenge is further complicated by the limited storability of electricity. Unlike commodity traders who can take physical delivery and warehouse their assets, most REPs must balance supply and demand instantaneously, 24/7/365. Our risk exposure includes:

• Price risk: Exposure to wholesale market volatility while serving fixed-price retail contracts

• Volume risk: Uncertain customer consumption patterns driven largely by weather events

Some other risks that we won’t cover today (but will in future posts) include:

• Credit risk: Customer payment default potential

• Regulatory risk: Changing market rules and compliance requirements

• Basis risk: Price differentials between financial hedging instruments and actual delivery locations

The complexity compounds because these risks interact in non-linear ways. A heat wave simultaneously increases consumption (volume risk) and market prices (price risk), creating a multiplicative rather than additive exposure.

Quantifying Risk

The first challenge is defining what risk means and figuring out how to measure it. For Base, risk means the potential financial exposure our retail electricity business faces due to market volatility. Specifically, it represents our obligation to procure power for our customers through counterparties or directly from ERCOT at prices that may substantially exceed what we've contracted to charge our customers.

Traditional approaches tend to use simplistic worst-case scenarios represented by the following equation:

Where:

• duration is a time frame that an organization feels captures worst case risk appropriately

• max(price) in ERCOT is $5,000/MWh

• max(load) is the load of their portfolio they have to serve over the duration

This leads to excessive conservatism and inefficient capital allocation. We needed a more nuanced approach.

The key insight was recognizing that risk is looking at the tails of our distribution and isn’t very different to how we would look at our expected P&L. In this case the distribution is the P&L our REP could make in a variety of outcomes. The other insight was risk is computed over some timeframe, e.g. our risk tomorrow or our risk for the remainder of the year, which can lead to having multiple risk metrics that are tracked. This takes a single number and gives you a view on the business from multiple angles.

As our customers pay us a fixed rate for electricity we can reduce the modeling complexity by only looking at our cost to serve our customers and ignoring the revenue portion of the P&L equation:

Where:

• c is a single customer, t is a 15 minute time step (ERCOT bills us based on electricity usage and prices in 15 minute increments)

• load(c, t) is the load of a given customer, c, at time, t

• price(t) is the ERCOT wholesale price of electricity at time, t (there are other costs associated with delivering power but for simplicity we’ll ignore them)

We can generate a distribution of cost to serve numbers over different load and price combinations. From here we can convert this into actionable metrics that are common in financial risk management:

1. Value at Risk (VaR): The nth percentile of our potential daily obligation, representing our exposure in all but the most extreme n% of scenarios.

2. Conditional Value at Risk (CVaR): The average of all outcomes above the nth percentile, providing visibility into the severity of tail events.

We chose to look at the 95th or 99th percentile as a way for us to understand our tail risk. You may be wondering why we don't look at the worst case scenario, we’ll dive into that in the next section.

Why Not Model the Absolute Worst Case?

Three key reasons:

1. Statistical uncertainty: The 100th percentile is inherently unstable and cannot be reliably estimated from historical data

2. Decision paralysis: Optimizing for the absolute worst case leads to prohibitively expensive hedging strategies

3. Capital efficiency: The 99th percentile offers a practical balance between protection and performance

Simulation Framework: Monte Carlo with Historical Bootstrapping

Our initial implementation uses a Monte Carlo approach with random sampling from historical distributions:

1. We sample from historical summer load and price data, preserving the correlation structure between these variables

2. We scale historical customer load profiles to our projected customer base

3. We calculate the resulting distribution of daily obligations

4. We extract the 99th percentile and compare it to our risk threshold

The power of this approach comes from modeling the full distribution of outcomes rather than fixating on a single scenario. This gives us a much more nuanced view of our risk landscape.

Practical Application: From Mathematics to Market Decisions

Our risk framework translates directly into actionable hedging strategies. Each morning, we view updated risk metrics on a dashboard showing:

1. Current VaR and CVaR by load zone

2. Scenario analyses for extreme weather events

3. Hedge recommendations to maintain risk within defined thresholds

We’ve built on this mathematical model to create a risk system that allows us to easily quantify the impact of our decisions, e.g. how would the risk profile and expected P&L would change if we purchased more hedges.

In a future post we’ll discuss how we incorporate our batteries into our strategy and we’ll deep dive into why over hedging can be detrimental to a REP.

About the author

Zayn Hanif works on the Markets Team, building algorithms and models to use Base batteries to stabilize the grid and power our retail engine. Before Base worked as a Quantitative Developer at Citadel on both their US Power Trading Desk & European Natural Gas Trading Desk.

If you find this work interesting, join our team by clicking the button below.

See Careers

Base Power provides Distributed Energy Resources (DERs) on the Texas grid by managing a large fleet of home backup batteries. These resources support the grid by providing power during times of high demand.

Grid-scale batteries are seeing wider adoption due to falling battery costs and increased penetration of non-dispatchable generation such as wind and solar. Power supply and demand have to be matched at all times. For example, solar stops producing when the sun goes down which coincides with the time people arrive at home, turn on the AC, cook dinner, and plug in their EV. This has created the infamous duck curve, where natural gas peaker plants have to surge to match the increasing demand and decreasing supply. Batteries allow us to time-shift energy by charging when power is abundant and discharging when power is scarce.

Centralized grid-scale batteries must pass through a multi-year interconnection queue. Base, however, uses distributed grid-scale batteries to quickly deploy storage assets on the grid by locating batteries throughout the distribution infrastructure.

Having real-time visibility into the state of the system is a key challenge when managing a fleet of distributed batteries. To effectively operate the fleet of batteries we need to know how power flows to the home and from the grid, batteries, and optional onsite solar panels. The largest utilities in Texas participate in Smart Meter Texas, which only provides 15 minute power granularity, whereas Base has two second visibility into power flows.

Defining a Modern Telemetry Stack

Base Power interviews start with a working session. Interviewees are asked to present on how they’d design a system that’s relevant to our current challenges. This is a useful signal because it lets us see how they’d apply their knowledge to aspects of our business. My working session was to design a telemetry stack for a fleet of distributed batteries.

The main challenge is getting the data off the device attached to the member’s home and into the cloud. If we don’t have access to the member’s Wi-Fi we have to rely on often unreliable and expensive 4G backhaul. We don’t want to lose data if the network is down for an extended time or if the device loses power.

My proposal:

Store-and-forward the data locally on device, send over encrypted proto/gRPC to a service in the cloud, and then from there write to a timeseries database. On the server you can optionally enrich the data and dual-write or publish to additional subscribers that need up-to-date telemetry.

The Existing State

When I joined I learned about the state of Base’s existing telemetry system, which was written quickly in the early days of the company as a proof of concept. Our edge software is hosted on a Raspberry Pi that reads metrics from the battery over Modbus. Previously it would package those metrics into a JSON object and publish them over MQTT. An AWS IoT Core rule would then write them to AWS Timestream.

There were a number of problems with the existing stack. JSON is easy to use but has a very verbose wire format that costs a lot to send over 4G. The metrics had no interface definition or compile time checks and so could be modified by fat-fingering a key name in JSON. We were using Timestream’s single measure record format which limited us to 100 metrics per JSON message. And finally we had unpredictable and high costs from Timestream, with no cost observability to determine what was driving the cost.

The telemetry stack wasn’t ideal but worked well enough that rewriting it wasn’t our top concern, we had too many other pressing priorities. However, keeping the end goal in mind and using it to inform how we built other components of the system, we were able to do a piecemeal migration to a new telemetry stack over a 6-7 month period.

Secure Device to Cloud Communication

One of my first projects at Base was to design a way of configuring our gateways. The gateways are Raspberry Pi devices connected to the battery that run golang software in containers. They are the interface between the cloud and the battery—forwarding telemetry from the battery to the cloud and sending commands to charge and discharge the battery. We needed a declarative way to define the configuration of the system: whether it’s a subpanel or whole home backup, if the member has solar, etc.

I set up a new gateway API ECS task that exposes a configuration endpoint for gateways to contact. I configured mTLS to mutually authenticate the cloud service to the gateway and the gateway to the cloud service, sharing the same certs we use for connecting to MQTT. This laid the foundation for a secure connection from the device to the cloud.

SQLite on the Edge

Devices receive power commands from the cloud to charge, discharge, or follow home load. These were stored in memory on the gateway and could be lost during container restarts (such as during a software update). We wanted the commands to persist until they expired.

I added SQLite to our gateways and created a volume that attached to the containers for storing the database files. We write commands to the database upon receipt and the gateway checks for the latest command in the database when it first starts. This gave us structured, durable on-device storage.

Proto/gRPC Telemetry Publishing

In the fall we implemented support for dual ground mount batteries. This required us to talk to and receive telemetry from two batteries at once. For dual battery systems we’d need to add a new dimension `role` to the Timestream metrics to allow us to differentiate between the individual batteries. This was further complicated by the fact that we were already at the limit of the number metrics we could publish in a single JSON object.

This called for a rewrite of our telemetry publishing and ingestion. I defined a proto file for representing all of our device metrics. We already had the gateway API and so I wrote a new gRPC service endpoint for reporting telemetry to replace the IoT Core rule we were using. Its request object wraps the serialized telemetry proto and includes an enum indicating what type of telemetry is being reported. The server uses proto reflection to iterate over all proto fields and prepare them for ingestion into Timestream.

We had configured MQTT store-and-forward and needed to maintain that behavior to prevent us from losing telemetry during network interruptions. Fortunately we already had durable storage on-device in the form of SQLite. We wrote a little SQLite wrapper that first stores to the database, then publishes the telemetry to the cloud, and finally deletes the database row if that was successful. If not successful, a background process periodically looks for unpublished telemetry and attempts to send it to the cloud.

This allowed us to add additional device metrics that had been blocked by the 100 measure limit and quickly launch dual battery systems. We were now in full control over our telemetry stack.

Datastore Migration

Finally it was time to reconsider a new datastore for our timeseries data. Our AWS Timestream bill had grown quickly and offered very little in the way of observability, so we had no idea what was driving our costs. Furthermore it was lacking in features you’d expect from a modern database. Considering our use cases, I realized we would be better served by a data warehouse type application. We looked at BigQuery, Snowflake, Clickhouse, and others, but settled on BigQuery because I had the most experience with it which would allow for a quick migration.

We extended our Terraform to work with GCP and turned up a basic integration using Workload Identity Federation to authenticate our AWS resources with Google Cloud. We built our own schema syncer to evolve our table schemas in safe ways (no deleting columns, no changing column types) and integrated it into our CICD pipeline. With that all setup, we were able to start dual writing to Timestream and BigQuery.

We partitioned our telemetry table by time and clustered on the gateway ID to minimize query costs. A key goal in the migration was to have better cost observability, so I created a Grafana dashboard that queries the INFORMATION_SCHEMA tables to calculate query costs on a per-dashboard basis. I modified our query wrapper to read the gRPC service and method from the golang context and add them as tags as part of the BigQuery request so that I could also aggregate costs per service.

Summary

Over the course of 6-7 months we did a piecemeal rewrite of our entire telemetry stack. The foundations were laid as necessary as part of other development work and delivered incremental value along the way. We were successful in lowering our costs, increasing reliability, and improving observability for both query performance and cost.

We now have a telemetry system that continues to deliver two second power flow data and will scale as we exponentially grow our fleet of distributed grid-scale batteries.

About the Author

Andrew Hitchcock works on the fleet team which is responsible for managing our fleet of batteries and the software that runs on-device at members’ homes. Before Base he launched Amazon EMR, worked as an SRE at Google, and built BigQuery’s realtime storage optimization system. In his free time he larps as a rancher and enjoys spending time with his two longhorns.

If you want to work on projects like this from start to finish, join our team by clicking the button below.

See Careers