Senior Data Engineer: Build Core, Scalable Data Pipelines

Windfalldata; Denver, San Francisco (CO, CA); On-site; USD 130,000 - 180,000; Full time

Job summary

Windfall is seeking a Sr. Data Engineer to join our data team. You will design and build pipelines for massive datasets, taking them from inception to production and customer use, while collaborating with product and data science teams.

You will own end-to-end data infrastructure on GCP, implement scalable processing with Spark/Beam, and orchestrate pipelines with Airflow, balancing speed and quality in a fast-paced environment.

Qualifications

• 4-8 years of professional data engineering experience.
• Experience with Beam/Spark/Flink or MapReduce.
• Strong OO programming ability in a JVM language.
• Expert knowledge of distributed data processing.
• Familiarity with different datastores and appropriate usages.

Responsibilities

• Construct data pipelines to ingest and merge billions of entities into Windfall’s core data asset.
• Collaborate with data science to run ML models on large data sets.
• Build data services and applications to orchestrate and monitor data systems.

Skills

GCP

Java

Python

Kotlin

BigQuery

Spark

Airflow

Distributed data processing

Tools

GCP

BigQuery

Postgres

Scylla/Cassandra

Apache Beam

Apache Spark

Flink

Airflow

Job description

Windfall is seeking a Sr. Data Engineer to join our data team. You will design and build pipelines for massive datasets, taking them from inception to production and customer use, while collaborating with product and data science teams.

You will own end-to-end data infrastructure on GCP, implement scalable processing with Spark/Beam, and orchestrate pipelines with Airflow, balancing speed and quality in a fast-paced environment.