Databricks Data Engineer Lead: Streaming & ETL Expert

August Schell; Rockville (MD); On-site; USD 120,000 - 170,000; Full time

Job summary

August Schell is seeking a skilled Data Engineer to lead data management and optimization across batch and streaming workloads. You will architect and maintain ETL pipelines, optimize Databricks/Spark workflows, and ensure data quality and governance in a fast-paced environment.

Ideal candidates have strong Python/Scala/Java, experience with Cribl, Airflow, and real-time processing using Kafka/Flink, plus a BA/BS or higher in a related field and 4+ years of data engineering experience.

Qualifications

• 5+ years’ experience as a Data Engineer with data management and streaming focus.
• Proficiency with Python, Scala and/or Java.
• Expert Databricks and Apache Spark experience mandatory.

Responsibilities

• Data Management Leadership: oversee data at rest and streaming, guide the team.
• ETL Pipeline Development: design and maintain efficient ETL pipelines with data quality.
• Databricks Expertise: optimize processing with Spark-based solutions and integration.
• Streaming Data: manage streaming sources and real-time processing with Kafka/Flink.
• Data Transformation: route, transform, and enrich data with Cribl or similar tools.
• Data Optimization: performance tuning, indexing, compression for storage efficiency.
• Data Security and Compliance: collaborate on privacy and security requirements.
• Documentation: build docs for data processes and pipelines.

Skills

Python

Scala/Java

Databricks

Apache Spark

Kafka

Flink

SQL

NoSQL

Cribl

Airflow

Education

BA/BS in relevant field

MS degree with 2+ years experience

Tools

Databricks

Apache Spark

Airflow

Cribl

Snowflake

Kafka

Flink

Job description

August Schell is seeking a skilled Data Engineer to lead data management and optimization across batch and streaming workloads. You will architect and maintain ETL pipelines, optimize Databricks/Spark workflows, and ensure data quality and governance in a fast-paced environment.

Ideal candidates have strong Python/Scala/Java, experience with Cribl, Airflow, and real-time processing using Kafka/Flink, plus a BA/BS or higher in a related field and 4+ years of data engineering experience.