Lead Data Engineer — Scale AI-Driven Data Pipelines

UNAVAILABLE; Chicago (IL); On-site; USD 98,000 - 182,000; Full time

Benefits offered by this job

Flexible time off

Paid sick time

Comprehensive health coverage

401(k)

Tuition assistance

Job summary

UNAVAILABLE is seeking an experienced engineer to build and maintain data pipelines using Spark and SQL. This role focuses on high-volume data environments and offers the opportunity to work with state-of-the-art technologies like AWS and Databricks.

Ideal candidates will have substantial experience in Apache Spark, Scala, and Python. You will also mentor junior engineers and contribute to a team that prioritizes autonomy and impactful work.

Qualifications

• 5+ years of professional experience building and maintaining big data pipelines.
• Deep hands-on expertise in Apache Spark and distributed computing concepts.
• Fluent SQL skills to ingest complex use cases and tune queries.

Responsibilities

• Write scalable code using Spark (Scala/Python) and SQL.
• Build and optimize data connectors for external publishers.
• Participate in code reviews and mentor junior engineers.

Skills

Apache Spark

Scala

Python

SQL

AWS

Databricks

Education

S. in Computer Science, Computer Engineering, or a related field

Tools

Apache Airflow

Kafka

Job description

UNAVAILABLE is seeking an experienced engineer to build and maintain data pipelines using Spark and SQL. This role focuses on high-volume data environments and offers the opportunity to work with state-of-the-art technologies like AWS and Databricks.

Ideal candidates will have substantial experience in Apache Spark, Scala, and Python. You will also mentor junior engineers and contribute to a team that prioritizes autonomy and impactful work.