Databricks Data Engineer Lead: Streaming & ETL Expert

August Schell

Rockville (MD)

On-site

USD 120,000 - 170,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

August Schell is seeking a skilled Data Engineer to lead data management and optimization across batch and streaming workloads. You will architect and maintain ETL pipelines, optimize Databricks/Spark workflows, and ensure data quality and governance in a fast-paced environment.

Ideal candidates have strong Python/Scala/Java, experience with Cribl, Airflow, and real-time processing using Kafka/Flink, plus a BA/BS or higher in a related field and 4+ years of data engineering experience.

Qualifications

  • 5+ years’ experience as a Data Engineer with data management and streaming focus.
  • Proficiency with Python, Scala and/or Java.
  • Expert Databricks and Apache Spark experience mandatory.

Responsibilities

  • Data Management Leadership: oversee data at rest and streaming, guide the team.
  • ETL Pipeline Development: design and maintain efficient ETL pipelines with data quality.
  • Databricks Expertise: optimize processing with Spark-based solutions and integration.
  • Streaming Data: manage streaming sources and real-time processing with Kafka/Flink.
  • Data Transformation: route, transform, and enrich data with Cribl or similar tools.
  • Data Optimization: performance tuning, indexing, compression for storage efficiency.
  • Data Security and Compliance: collaborate on privacy and security requirements.
  • Documentation: build docs for data processes and pipelines.

Skills

Python
Scala/Java
Databricks
Apache Spark
Kafka
Flink
SQL
NoSQL
Cribl
Airflow

Education

BA/BS in relevant field
MS degree with 2+ years experience

Tools

Databricks
Apache Spark
Airflow
Cribl
Snowflake
Kafka
Flink

Job description

August Schell is seeking a skilled Data Engineer to lead data management and optimization across batch and streaming workloads. You will architect and maintain ETL pipelines, optimize Databricks/Spark workflows, and ensure data quality and governance in a fast-paced environment.

Ideal candidates have strong Python/Scala/Java, experience with Cribl, Airflow, and real-time processing using Kafka/Flink, plus a BA/BS or higher in a related field and 4+ years of data engineering experience.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks SME
Databricks SME

August Schell • Rockville (MD)

On-site
USD 120,000 - 170,000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000
Data Engineer
Data Engineer

GSquared Group • Atlanta (GA)

On-site
USD 120,000 - 160,000
Principal/Lead Data Engineer
Principal/Lead Data Engineer

Accion Labs • United States

On-site
USD 180,000 - 260,000
Lead Data Engineer: Databricks & Azure Streaming Architect
Lead Data Engineer: Databricks & Azure Streaming Architect

84.51˚ • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Health insurance
401(k) with matching
Paid time off
Databricks Data Engineer Lead | ETL & Data Lake Expert
Databricks Data Engineer Lead | ETL & Data Lake Expert

ContractStaffingRecruiters.com • Branford (CT)

Remote
USD 80,000 - 110,000
Senior Data Engineer
Senior Data Engineer

Compunnel Inc. • Sunnyvale (CA)

On-site
USD 100,000 - 140,000
Databricks Architect -- KATDC5738914
Databricks Architect -- KATDC5738914

Compunnel Inc. • Edison (NJ)

On-site
USD 100,000 - 130,000
Lead Data Engineer: Streaming Data on Azure & Databricks
Lead Data Engineer: Streaming Data on Azure & Databricks

84.51° • Cincinnati (OH), Chicago (IL)

On-site
USD 125,000 - 207,000
Databricks Data Engineer
Databricks Data Engineer

Compunnel Inc. • Spring (TX)

On-site
USD 110,000 - 140,000