Databricks Data Engineer Lead — ETL, Streaming & Spark

AUGUST SCHELL ENTERPRISES, INC.; Rockville, Northern (MD, KY); Hybrid; USD 110,000 - 160,000; Full time

Job summary

AUGUST SCHELL ENTERPRISES, INC. in Rockville, MD, is seeking a highly skilled Data Engineer to manage data at rest and in streaming, and to lead data management and optimization initiatives.

You will shape ETL pipelines, leverage Databricks, and implement Spark-based solutions to drive data integration and analytics. Ideal candidates bring 5+ years of data engineering experience, expert knowledge of Databricks and Spark, and hands-on skills in streaming tech and data security.

Qualifications

• 5+ years’ experience as a Data Engineer with data management and streaming expertise.
• Proficient in Python, Scala and/or Java for ETL and data processing.
• Consultant-level, expert experience in Databricks.
• Apache Spark experience (mandatory).

Responsibilities

• Lead data management initiatives for data at rest and streaming.
• Design, develop and maintain ETL pipelines ensuring data quality.
• Optimize Databricks Spark-based processes and data integration.
• Manage streaming data sources and real-time processing.
• Transform, route, and enrich data with Cribl or similar tools.
• Improve data storage performance via indexing and compression.
• Collaborate with security teams on data privacy and compliance.
• Create documentation and enable knowledge transfer.

Skills

Data management

ETL pipelines

Python/Scala/Java

Databricks

Apache Spark

Streaming data

SQL/NoSQL

Data security

Education

Bachelor's degree in CS/CE/Math

Master's degree in CS

Tools

Cribl

Airflow

Snowflake

Job description

AUGUST SCHELL ENTERPRISES, INC. in Rockville, MD, is seeking a highly skilled Data Engineer to manage data at rest and in streaming, and to lead data management and optimization initiatives.

You will shape ETL pipelines, leverage Databricks, and implement Spark-based solutions to drive data integration and analytics. Ideal candidates bring 5+ years of data engineering experience, expert knowledge of Databricks and Spark, and hands-on skills in streaming tech and data security.