Data Engineer

Consensus; San Francisco (CA); On-site; USD 160,000 - 230,000; Full time

Benefits offered by this job

Medical insurance

Vision insurance

401(k)

Paid maternity leave

Paid paternity leave

Job summary

A leading AI research firm in San Francisco is seeking a skilled Data Engineer to design and maintain data pipelines that underpin their product. Ideal candidates should have over 3 years of experience in building production data pipelines, along with strong skills in SQL and Python. This position offers a competitive salary range and equity in a rapidly growing startup focusing on research and science. Join a mission-driven team to propel the future of research.

Qualifications

• 3+ years building production data pipelines.
• Strong SQL + Python skills required.
• Experience with distributed data processing tools like Databricks or Spark.

Responsibilities

• Build and maintain production pipelines for data.
• Clean and unify raw data into consistent datasets.
• Ensure data quality through validation and monitoring.

Skills

Production data pipelines

SQL

Python

Distributed data processing

Data validation practices

Tools

Databricks

Spark

Dagster

GitHub

Job description

Get AI-powered advice on this job and more exclusive features.

This range is provided by Consensus. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

Base pay range

$160,000.00/yr - $230,000.00/yr

Consensus is building the OS for research. Today, our academic search engine helps 8 million students, researchers, and doctors analyze scientific papers and complete literature reviews 10x faster.

Our Series A was led by USV, with major participation from top AI investors, including Nat Friedman and Daniel Gross. Consensus has been featured in The Wall Street Journal, The Atlantic, The New York Times, Nature, and a16z as one of today’s most exciting and important AI startups.

Our mission is to empower the world to understand, create, and apply good science. Help us build the future of research.

Role: Data Engineer to design, build, and maintain the data pipelines and infrastructure that form the foundation of our product.

Responsibilities

• Build and maintain production pipelines for third‑party data (APIs, files, partner feeds).
• Clean, normalize, and unify raw into clear, consistent tables/datasets.
• Handle de‑duplication / entity resolution so records are canonical and trustworthy.
• Own data quality: validation, monitoring/alerting, and documentation.
• Partner closely with ML/product/backend to design schemas and ship data features that unblock use cases.
• Optimize pipelines for performance and cost, and improve CI/CD + testing for data jobs.

Qualifications

• 3+ years building production data pipelines.
• Strong SQL + Python.
• Experience with distributed data processing (Databricks/Spark or similar).
• Experience with an orchestrator (Dagster/Airflow/Prefect); Dagster is a plus.
• Proven ability to ingest from multiple sources and produce clean, well‑modeled datasets.
• Familiar with GitHub + CI/CD and data observability practices (validation, logging, monitoring, error handling).
• High ownership, fast execution, and good product instincts around how data impacts UX.
• Interest in the product and mission: science, research, education

Compensation:

• $160-$230k cash
• Competitive Series A equity

Final offers are determined by multiple factors and may vary from the amounts listed above.

Seniority level: Mid‑Senior level

Employment type: Full‑time

Job function: Software Development

Benefits:

• Medical insurance
• Vision insurance
• 401(k)
• Paid maternity leave
• Paid paternity leave

Location: Mountain View, CA