Senior Data Engineer - Spark & Airflow (Remote)

People Data Labs; Oregon; Hybrid; USD 200,000 - 220,000; Full time

Benefits offered by this job

Stock

401k

Competitive Salaries

Unlimited paid time off

Medical, dental, & vision insurance

Health, fitness, and office stipends

Remote-friendly work environment

Job summary

People Data Labs seeks a Senior Data Engineer to design and scale data pipelines that process hundreds of billions of entities, leveraging Spark, Python, and cloud infrastructure. You will contribute to an entity-resolution framework and data lakehouse patterns to deliver clean, actionable datasets.

You’ll thrive with ownership and collaboration in a remote-friendly environment, using Airflow, dbt, and Delta Lake alongside Snowflake/Redshift BigQuery integrations, while helping shape data design

Qualifications

• 5-7+ years of industry experience with clear examples of strategic technical problem-solving and implementation.
• Strong software development fundamentals.
• Experience with Python.
• Expertise with Apache Spark (Java, Scala, and/or Python-based).
• Experience with SQL.
• Experience building scalable data processing systems from the ground up.
• Experience using Airflow, dbt, dagster or similar.
• Knowledge of modern data design and storage patterns.
• Experience working in Databricks (Delta Live Tables, lakehouse patterns).
• Experience with cloud computing services (AWS, GCP, Azure).
• Experience with data warehousing (Databricks, Snowflake, Redshift, BigQuery).
• Understanding of modern data storage formats (Parquet, ORC, Avro, Delta Lake).

Responsibilities

• Build infrastructure for ingestion, transformation, and loading an exponentially increasing volume of data from a variety of sources using Spark, SQL, AWS, and Databricks
• Building an organic entity resolution framework capable of correctly merging hundreds of billions of individual entities into a number of clean, consumable datasets.
• Developing CI/CD pipelines and anomaly detection systems capable of continuously improving the quality of data we're pushing into production.
• Dreaming up solutions to largely undefined data engineering and data science problems.

Skills

5-7+ years experience

Software development fundamentals

Python

SQL

Scalable data processing

Airflow

dbt

Dagster

Education

Quantitative degree (CS/Math/Stats/Engineering)

Tools

Apache Spark

Databricks

AWS

GCP

Azure

Snowflake

Redshift

BigQuery

Delta Lake

Kafka

Job description

People Data Labs seeks a Senior Data Engineer to design and scale data pipelines that process hundreds of billions of entities, leveraging Spark, Python, and cloud infrastructure. You will contribute to an entity-resolution framework and data lakehouse patterns to deliver clean, actionable datasets.

You’ll thrive with ownership and collaboration in a remote-friendly environment, using Airflow, dbt, and Delta Lake alongside Snowflake/Redshift BigQuery integrations, while helping shape data design