Firestore to BigQuery NRT Connector

Near-real-time data sync processing 2M+ records/day at 99.9% reliability with 60-70% compute cost reduction.

Overview

Built a serverless, event-driven pipeline that syncs Firestore document changes to BigQuery in near-real-time. The system uses Eventarc to capture Firestore triggers, Cloud Functions for transformation, and Pub/Sub dead-letter queues for fault tolerance, all provisioned via Terraform.

This replaced a batch-based approach that had hours of latency and significantly higher compute costs.

Architecture

flowchart TD
    A[("Firestore\nCDC")] --> B[Eventarc]
    B --> C["Cloud Function\ntransform + validate"]
    C --> D[("BigQuery\nstreaming insert")]
    C -->|on failure| E["Pub/Sub DLQ"]
    E --> F["Cloud Function\nretry handler"]
    F --> D

Tech Stack

Layer	Technology
Source	Firestore (document DB)
Trigger	Eventarc
Processing	Cloud Functions (Python)
Storage	BigQuery
Error Handling	Pub/Sub Dead Letter Queue
IaC	Terraform

Key Metrics

Metric	Value
Throughput	2M+ records/day
Reliability	99.9%
Compute cost reduction	60-70% vs. batch approach
Latency	Near-real-time (seconds)

Challenges & Decisions

Why serverless over Dataflow?

Problem: Initial design considered Dataflow for streaming, but the event volume didn’t justify always-on workers.
Approach: Cloud Functions with Eventarc triggers - pay-per-invocation, zero idle cost.
Outcome: 60-70% cost reduction while maintaining sub-minute latency.

Dead-letter queue for fault tolerance

Problem: Transient BigQuery streaming insert failures could lose records.
Approach: Pub/Sub DLQ captures failed events; a separate Cloud Function retries with exponential backoff.
Outcome: 99.9% end-to-end reliability with zero manual intervention.

Terraform for reproducibility

Problem: Multiple environments (dev, staging, prod) needed identical infrastructure.
Approach: Full IaC with Terraform modules - one terraform apply provisions the entire pipeline.
Outcome: Environment parity, auditable infrastructure, fast rollbacks.

What I Learned

Serverless event-driven architectures can match Dataflow throughput at a fraction of the cost for mid-volume workloads.
DLQ patterns are non-negotiable for production NRT systems - silent data loss is worse than latency.
Terraform modules pay for themselves the moment you need a second environment.

Built at Wayfair India · Dec 2024 – Present