Software Engineer, Data Infrastructure (Staff)

Doist

San Francisco (CA)

On-site

USD 180,000 - 280,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Benefits offered by this job

Competitive salary

Meaningful early equity

Health insurance (medical, dental, and

3 weeks of PTO

11 paid company holidays

3 months of paid family leave

Wednesdays work from home

401k plan

Commuter and lunch stipend

Job summary

Lightfield is hiring a Staff Engineer, Data Infrastructure to build the next generation of data systems. The role spans backend, infrastructure, and product-facing data challenges, with a focus on scalable analytics and high-volume data processing.

You will work on Postgres-backed systems, Redis-backed queues, and per-organization access controls while collaborating with product and engineering teams to scale the platform.

Qualifications

Strong software engineering fundamentals with production data systems experience.
Ability to optimize query plans, data freshness, and correctness with user-impact.
Experience designing scalable data architectures and supporting product/engineering velocity.

Responsibilities

Scale the analytics engine behind customer dashboards and ensure performance under security constraints.
Design ingestion paths, event models, and schemas for data movement from writes to search and dashboards.
Evolve a schema-flexible, graph-shaped data model for fast querying as data grows.
Build foundations for historical reporting and auditability, including change capture.
Develop data systems for usage metering, pipeline generation, and AI evaluation with business impact.
Decide when current architecture suffices and when new systems are warranted.
Set technical direction and engineering practices for data systems as the company scales.

Skills

Production data systems

Query performance

Schema design

Data infrastructure

Cross-functional collaboration

Tools

Postgres

Redis

Typesense

BullMQ

SQL

Observability tools

Job description

About Lightfield

Lightfield is an AI-native CRM that assembles itself from your email, calendar, and meetings. It captures every interaction and turns it into organized context: accounts, tasks, follow-ups, and insights, so nothing slips through the cracks. We’re rethinking CRM from first principles. Instead of forcing teams to maintain rigid systems, Lightfield learns from how companies actually work, adapting, automating, and surfacing the insight that drives growth. We’re building the CRM platform we always wished existed: fast, intelligent, and genuinely helpful. We are backed by Greylock, Lightspeed, and Coatue, and our founders previously built Tome, a generative AI presentation product used by over 25 million people. Before Lightfield, our team worked on Llama, Instagram, Facebook Messenger, Pinterest, Google, and Salesforce.

About the role

We’re building the infrastructure foundation for a fast-growing AI product company serving thousands of customers: Our Postgres fleet serves 5B+ queries a month — roughly 2,500 QPS steady state, with sustained spikes past 25,000 QPS — and database workload more than doubled last month. Redis sustains ~50,000 commands per second behind a job platform that executes 10M+ background job runs a day across ~170 queues. We ingest tens of millions of emails and calendar events a month. That growth creates scaling pressure across backend systems, infrastructure, and data infrastructure. We’re hiring a staff-level engineer who spikes in data infrastructure but is excited to work across backend systems, infrastructure, and product-facing data problems. The work is close to the product, close to customers, and close to production.

As a Software Engineer, Data Infrastructure, you’ll build the next generation of our data systems. We got remarkably far on a deliberately simple stack: Postgres as the system of record, a sharded transactional outbox for change events, Redis-buffered sync into Typesense for search, BullMQ for processing, and Postgres-backed customer-facing analytics with per-organization row-level security.

The next phase is evolving that pragmatic foundation into best-practice data architecture: change data capture, event modeling, schema design, query performance, freshness guarantees, and the right boundary between transactional and analytical workloads.

The system of record itself is unusual. Customers define their own objects, attributes, and relationships at runtime, so the core data model is a schema-flexible, graph-shaped store: entity-attribute-value with typed edges, versioned attribute values, and relationship history. That makes schema design, indexing, and query performance genuinely hard problems rather than routine tuning.

The surface area is wider than analytics: customer-facing dashboards, historical and audit data, datasets that power pipeline-generation products, and evaluation data that measures our AI agents. This is data infrastructure work, not a BI or dashboarding role. It’s a good fit for someone who likes high-volume data systems, pragmatic architecture decisions, and building foundations that product and engineering teams can actually depend on.

This role can be based in San Francisco or Cambridge. In San Francisco, you’d work from our HQ alongside the founders and most of the engineering team. In Cambridge, you’d join an initial group of staff-level engineers at our new, infrastructure-focused Kendall Square site, working alongside one of our most senior infrastructure engineers. We aim to build the site and organization around this group as the company scales.

What you’ll do

Scale the analytics engine behind customer-facing dashboards, tackling query performance under row-level security, workload isolation, read architecture, and observability as data volume grows.
Design the ingestion paths, event models, schemas, and query patterns that move data from transactional writes into search, dashboards, and history—with clear guarantees around freshness, correctness, replay, and failure recovery.
Evolve our schema‑flexible, graph‑shaped data model so customer‑defined objects, attributes, and relationships remain fast to query as their size and complexity grow.
Build the foundations for historical reporting and auditability, including attribute versioning, relationship history, and change capture.
Build reliable data systems for usage metering, pipeline generation, and AI evaluation, where errors have direct customer, product, or financial consequences.
Decide when our existing architecture remains the right foundation and when new analytical, streaming, or workflow systems earn their added complexity.
Set the technical direction, abstractions, ownership boundaries, and engineering practices for data systems as the company grows.

What your first year looks like

Scaling the analytics serving path behind customer-facing dashboards is the anchor project, but the work stays close to the product. The current slate also includes:

Zero-downtime schema migrations for an 18-collection Typesense search deployment.
A usage‑metering pipeline for consumption billing.
Historical and audit data modeling.
Evaluation data infrastructure for our AI agents.

What we’re looking for

Strong software engineering fundamentals. Experience owning production data systems where query plans, replication lag, backfills, data freshness, schema evolution, or data correctness had real user‑facing consequences. Comfort debugging across multiple layers of the stack. Good judgment about when to make a tactical fix and when to invest in a more durable platform or architecture change. Product orientation: you care about how data infrastructure decisions affect customers, users, and engineering velocity. Clear communication, strong ownership, and a bias toward practical tradeoffs.

Helpful experience

You do not need all of these:

ClickHouse, OLAP systems, event pipelines, data warehouses, or analytical infrastructure.
Kafka, Flink, Spark, Iceberg, or similar streaming and lakehouse systems.
Postgres at scale, and the boundary between OLTP and OLAP systems.
APIs, queues, workflow systems, and distributed systems.
Observability, incident response, service ownership, and production debugging.
Data for ML/AI systems: enrichment pipelines, eval harnesses, or data‑quality tooling.
Experience in a high-growth product environment.

Why this role is interesting

You’d be building our analytical data architecture from close to the beginning — the foundations are deliberately simple, and the architecture that scales them is yours to shape. Customer-facing data products are on the roadmap, database workload more than doubled last month, and the foundations you build will carry the company for years.

Benefits & Perks