Adding a field to a database document shouldn’t also be a decision about what users can see in their browsers.
With our frontend listening directly to Firestore, it could be. The browser received whole documents, permissions had to stay synchronized between Firestore rules and our Go backend, and features that needed data from other databases ran into those boundaries. We needed the backend to decide what each user received and keep that decision up to date as both the data and their permissions changed. At millions of live subscriptions, we also needed to do it efficiently.
That led us to build Lovable Realtime, our own realtime service.
I built the core in 40 hands-on working days, and we rolled it out to every user without a single user-facing incident. The service handles peaks of over 2 million concurrent subscriptions and delivers around 2.6 billion data and patch frames per month.
We needed three things together: support for our existing databases, backend-shaped payloads, and reactive authorization. The alternatives either required changing our database stack or left substantial parts of that framework for us to build. Here’s how we evaluated those trade-offs, built the service, and tested the migration before switching users over.
Challenges with Firestore
Historically, parts of Lovable handled real-time updates by opening Firestore onSnapshot listeners straight from the browser. This has a few problems. It creates two authorization paths: Firestore rules were a mirror of our API’s permissions model in Golang, and we constantly had to remember to keep them in sync. In addition, the browser reads the entire Firestore document on each call. Anyone adding a field to the database is deciding what is exposed to the browser, while not necessarily knowing it. This was a latent but serious risk.
Finally, Firestore’s authorization rules are limited in that they can only query Firebase data. This was a blocker for us when we set out to enable authorization updates for large groups (e.g., the ability to give everyone in your company’s engineering group access to a project). Enterprises use our SCIM feature that syncs users and groups to their Lovable workspace. Features like SCIM need large DB transactions and complex query patterns, which document DBs such as Firestore traditionally do not provide to the same level of a solution like Spanner. But Firestore Security Rules can’t read Spanner.
We rolled out a temporary workaround: proxying onSnapshot subscriptions through our Go API only for businesses and enterprises. This reused the API’s permission system, but the proxy opened separate Firestore listeners for each client’s data dependencies.
At millions of live subscriptions, this would mean maintaining millions of long-lived upstream streams, often watching the same data repeatedly. Those streams also consumed capacity in the API’s shared Firestore connection pool, competing with ordinary database requests. A separate pool would isolate that contention, but would not solve the per-client listener growth. We knew this was a temporary fix and began searching for a new solution.
Requirements & options considered
As we looked to move away from Firestore, we had three main requirements. First, our data was in Firestore and Spanner and we didn’t want to migrate to a new primary database—the solution would need to work with the data stores we already had (and allow us to add new ones over time). Second, we need authorization to be re-evaluated continuously; it should be able to change while a connection is open and a revoked user’s stream should close mid-session. And third, the backend needs to control the permissions and shape the payload in server code; fields a caller lacks permission for should never leave the server.
We found options for each requirement, but none of the solutions we evaluated met all three requirements while working with our existing data store and Go backend.
Transport services such as Ably, Pusher, PubNub, and Centrifugo provide connections, delivery, and revocation mechanisms. We would still need to build change detection, affected-subscription tracking, and authorized projections.
Replicache lets us bring our own backend: a notification prompts the client to pull, and the server computes its authorized view. We would still need to implement notification routing, server-side views, and change tracking, including changes to the payload caused solely by permission changes.
Database-integrated systems such as Firestore, Firebase RTDB, Supabase Realtime, Hasura, PowerSync, and Zero offer different combinations of live queries, projections, and authorization within their supported stacks. We needed a database-agnostic layer over our existing Firestore and Spanner data, with results computed in our Go backend and access re-evaluated when authorization dependencies change. These products did not provide that combination without substantial integration work.
Electric syncs subsets of Postgres data to clients. Using it would have meant moving or replicating our Firestore and Spanner data into Postgres, rather than building directly on our existing stores.
Meteor’s publications shape per-user payloads in server code, with reactivity centered on MongoDB. Permission dependencies read separately from the reactive data need explicit handling. Convex is close to what we wanted: a server function becomes a reactive, server-shaped subscription, but it requires using Convex’s database.
We built the framework that connects data dependencies, permission changes, and backend-computed subscription results, without moving our data to a new store.
What we built
Lovable Realtime is a dedicated WebSocket service that streams live data, with authorization and data shape handled server-side in Go. We have one WebSocket per tab, with all subscriptions multiplexed over that connection. Database changes flow through change data capture (CDC), which automatically triggers affected subscriptions to recompute and push updates. Code writing to the database does not need to trigger subscription updates manually. The backend controls what each user can see and which fields are redacted, using the same permissions framework as our API.
The realtime service runs in Go on GKE. Each pod holds a single NATS connection and thousands of WebSocket connections. Data change events are sent from Firestore and Spanner into a bridge and then published to NATS. When a NATS message arrives, the pod dispatches it to every matching subscription instance, which re-evaluates its data projection, deduplicates via xxh3-128 hash, and pushes the deltas to the frontend if the output changed.
NATS for pub/sub
The service uses NATS as the message bus, which is similar to Redis pub/sub or Valkey pub/sub but has some advantages. For one, it supports hierarchical wildcard subjects (rt.data.workspace.W.project.*) that are matched server-side in a trie. Redis PSUBSCRIBE scans the pattern list on every publish, and sharded pub/sub drops pattern support entirely.
With NATS wildcards, the publisher writes to one specific subject and subscribers pick their own granularity. This works well for our use case where we have ~40 subscription types owned by 7 teams. It avoids the problem of multi-publishing to roll-up channels, where the publisher has to know every granularity anyone will ever subscribe at, and the roll-up channels themselves can quickly become overwhelmed. A workspace-wide channel for example can concentrate every event in that workspace onto one shard, in sharded Redis.
NATS also supports interest-based cluster routing for scalability: messages only go to nodes that have a matching subscriber, whereas classic Redis cluster pub/sub broadcasts every message to every node.
Developer experience
There’s a backend framework and a frontend SDK. In the frontend, you just need to give the name of the subscription you want to subscribe to and a parameter:
const { data, status, error } = useSubscription("project", { projectId });
This is fully typed end-to-end, with params and payloads generated from Go structs. Multiple components subscribing to the same thing will share the same server-side subscription automatically. The SDK also handles reconnect with backoff (and jitter) and sends auth token refresh frames to the server.
On the backend, you write a subscription function like you’d write an endpoint: get the params, make sure they have access, convert it to the response object you want, and return it.
Simplified pseudocode
func projectSub(call) {
p := call.Params
perms := requireAccess(call, ViewProject)
project := get(Project, p.projectID)
return toDTO(project, perms.has(ViewProjectUsage))
}
Under the hood, whenever any data you used in this function changes, the framework will call your function again and return new values. The output will be hashed against the previous output and deltas are sent to the frontend if it’s different.
A subscription can query Firestore and Spanner and combine their results into one live view, exposing only the fields the caller can access. When the underlying data or permissions change, the framework recomputes that view and pushes any changes to the client.
For observability, every subscription must declare an owning team or it will fail to register. Alerts route to whoever owns the data, with each subscription treated like an API endpoint.
Reactive data sources
So what is the “project” we are getting in the previous example? This is what we call reactive data sources or RDSes, which are the backbone of subscriptions.
Simplified pseudocode
Project = RDS[Project]{
Query: func(ctx, p) {
return db.GetProject(p.id)
},
Subjects: func(p) {
return []string{"rt.data.project." + p.id}
},
Apply: func(cached, ev) {
if ev.deleted {
return nil
}
return decode(ev.data)
},
TTL: 5 * time.Minute,
}
As you can see in the code above, an RDS is basically an abstraction of 3 things:
How to get some data (a query function)
Where to get updates for that data (NATS subjects)
What to do when you get updates from that data (this is the apply function, which acts like a reducer that reconciles what has changed)
Each data source can also set a TTL: a limit on how long cached data can be reused without querying the source again. Past the TTL, the cached snapshot stops being served, and the next read runs the query against the source of truth again.
This example is really simple (a point lookup that listens for one thing), but you can create complicated RDSes as well, like a “top 10 most visited projects in a workspace.”
Authorization is another RDS
Authorization itself is just another RDS! It calls the authorization provider, gets the permission set, and listens to changes on settings, ACLs, etc.
At first, we recomputed permissions whenever documents involved in an access decision changed, including the project document. But project documents change on every file edit, so every update to a project subscription triggered a full permissions recompute.
The fix was to compare the incoming document with the cached one and recompute only when a field that feeds an access decision changed. This led to about 84% fewer recomputes. We keep explicit field comparisons for permission evaluation, and a static-analysis test checks for missing and stale entries within the permission decision packages it covers. These comparisons are part of the authorization logic and need the same care as the permission checks themselves.
Subscription simulator
Start with the workspace example: subscribe, rename the workspace, change permissions, or update an internal field, and see what reaches the client. Then try a live query for the newest 10 projects, or a project overview that combines data from Spanner and Firestore. The simulator shows how snapshots, patches, deduplication, and reconnect reuse work.
Open the interactive realtime simulator in a new tab.
Rolling it out and running it
We used shadow traffic, then gradually enabled the service over about a week. In shadow mode, the frontend subscribed to the new service, exercised the full path, and discarded the payload while it continued rendering from Firestore. This gave us a real picture of the traffic and data our service would see, along with edge cases like tabs that stay open for nine days.
This shadow launch helped us find and fix multiple issues, including reconnect storms after deploys, misclassified errors, and the fact that we had a payload cap that was too small for real data.
We did an experimental rollout behind a feature flag, scaling from 1%, to 10%, to 50%, and finally to 100% of users over several days. The rollout was incident-free. We observed no regression in the monitored rollout metrics.
Running long-lived WebSocket connections on Kubernetes changes the usual scaling assumptions. For one thing, round-robin load balancing is wrong for sockets that live for minutes to hours, so we balance on connection count instead. Scale-out also doesn't relieve a hot pod, because connections are sticky. We deliberately limit connections per pod so that losing or replacing one pod does not trigger a huge reconnect wave. We reserve memory headroom and scale on connection count, CPU, and memory, with conservative scale-in to limit reconnect churn.
Every deploy is a mass-disconnection event, so the deploy machinery is really herd management. We canary in 10% steps with ~4-minute pauses, gated on WebSocket upgrade success rate, upgrade latency, pod restarts, and subscription server-fault rates. The frontend SDK spreads post-deploy reconnects over 20 seconds. On reconnect, the client sends a fingerprint of its cached state. After rechecking authorization and loading current data, the server can confirm that state is unchanged, avoiding retransmission and unnecessary client rerenders.
Results & next steps
As a result of this change, Firestore is no longer used by the frontend at all.
The following measurements cover Sep 5, 2026–Oct 5, 2026, with both boundaries at 00:00 UTC. Peak concurrent live subscriptions reached 2.06 million. The service delivered 2.61 billion data and patch frames over the month, averaging 86.9 million per day; resumed control frames are excluded.
Over this 30-day period, sending every computed result as a full snapshot would have produced an estimated 40 TB of application payload. Hash-based deduplication suppressed 88.4% of payload evaluations. Accounting for differences in payload size, we estimate that this avoided about 80% of the bytes, reducing the full-snapshot baseline to 8.20 TB.
For changed updates, sending deltas reduced payload volume by 94.5%. With full snapshots still used for initial subscriptions and reconnects, the total was 2.24 TB. Reusing unchanged state on reconnect reduced the actual payload sent to 1.11 TB—an estimated 97% reduction overall compared with sending every computed result as a full snapshot. These figures measure application payloads and exclude protocol and transport overhead.
Subscription-open success was 99.85% across all recorded outcomes, and 99.9975% excluding expected rejections and connection cancellations. Separately, WebSocket handshake success was 99.93% across all outcomes and 99.99916% excluding expected rejections.
Update latency from the source database write to the frame being ready to send was 66 ms at p50, 178 ms at p95, and 249 ms at p99. These figures exclude socket write, network, and client processing.
The biggest win, though, is qualitative: the backend now owns authorization and data shape. Access is re-checked continuously against centralized, auditable rules, and the enterprise features that were blocked on Firestore’s rule engine now just work for everyone.
We have a few takeaways for teams building something similar:
Use shadow traffic before switching users over: Shadow mode was the only way for us to generate load that truly reflected real-world traffic, data shapes, and user behavior, and we feel that’s what led to a successful rollout.
Choose weak delivery guarantees on purpose: We made the architecture simple by discarding any guarantees that we didn’t need. For our UI snapshots, we didn’t need ordered or exactly-once delivery. Ultimately TTL and per-subject watermarks (which let us ignore any data that arrives out of order) were all we needed.
Make the framework's simple path the whole path: With our SDK, authors write a function as if they were writing an API endpoint, consumers write two lines, and everything else is the framework's job. Where possible, we also don’t rely on people (or agents) to remember to do something correctly—everything is typed end-to-end from Go structs and we have CI checks to catch any drift. Today there are ~40 subscription types owned by seven teams, and people ship their own without any help.
Snapshots all the way down, deltas only at the wire: Every layer of the framework—hydration, event merges, the subscription function, hash dedup—deals in data snapshots. A snapshot becomes a delta only at the last step before the socket. This means that we get the wire savings described earlier, but the full data snapshot works as a single representation to reason about everywhere (and can be used to heal any frontends that get into a bad state).
Give coding agents a test harness that exercises the whole system: Our integration rig includes Firestore, Auth, and Spanner emulators plus an embedded NATS server, with real ID tokens minted by the Auth emulator. I worked through the architecture and edge cases, then delegated most of the implementation to coding agents. The harness let them test their changes across the full stack.
In terms of what’s next: The framework is optimized for data snapshots, but there are times when you just want to know that some event happened (and don’t care about the data snapshot). This is possible with the RDS primitive, but a bit awkward, so we’re actively working on adding domain event primitives to the architecture.
The frontend no longer depends on Firestore. Teams can build richer realtime collaboration while the backend controls authorization and payloads, regardless of how our database choices evolve. The service could eventually become a capability available in Lovable-built apps.
If you’re interested in working on projects like this at Lovable, check out our open roles—we’re hiring!
