Databricks Data Engineer — Streaming & ETL Lead

augustschell; Rockville (MD); On-site; USD 120,000 - 180,000; Full time

Job summary

August Schell is seeking a highly skilled Data Engineer to lead data management, optimize ETL pipelines, and drive streaming data initiatives in a dynamic environment. You will work across Databricks, Spark, and Snowflake to deliver robust data solutions and ensure data quality and security.

Ideal candidates bring 5+ years in data engineering, strong Python/Scala/Java coding, and hands-on experience with streaming and data transformation tools.

Qualifications

• 5+ years’ experience as a Data Engineer with a deep understanding of data management, ETL pipelines and streaming data.
• 5+ years’ experience writing code using Python, Scala and/or Java.
• Consultant-level, expert experience in Databricks.
• Apache Spark experience (mandatory).
• 2+ years’ expertise in Databricks, Apache Spark, Snowflake and/or similar data processing frameworks.

Responsibilities

• Data Management Leadership: Provide technical leadership in data management, overseeing data at rest and data streaming, and guiding the team.
• ETL Pipeline Development: Design, develop, and maintain efficient ETL pipelines to process and transform data, ensuring data quality, integrity, and consistency.
• Databricks Expertise: Optimize data processing by using your expertise in Databricks, develop Spark-based solutions, and enhance data integration and analytics capabilities.
• Streaming Data: Manage streaming data sources and implement real-Ime data processing solutions using tools like Apache Kana or similar technologies.
• Data Transformation: Implement and manage data routing, transformation, and enrichment using Cribl or similar data pipeline orchestration tools.
• Data Optimization: Work on data optimization initiatives, including performance tuning, indexing, and data compression, to ensure efficient data storage and retrieval.
• Data Security and Compliance: Collaborate with security teams to ensure data security and compliance with data privacy regulations.
• Documentation: Build comprehensive documentation for data management processes, ETL pipelines, and facilitating knowledge transfer within the team.

Skills

Data management

ETL pipelines

Python/Scala/Java

Databricks

Apache Spark

Streaming data

Snowflake

SQL/NoSQL

Data security

Education

BA/BS or MS in relevant field

Tools

Databricks

Apache Spark

Cribl

Airflow

Job description

August Schell is seeking a highly skilled Data Engineer to lead data management, optimize ETL pipelines, and drive streaming data initiatives in a dynamic environment. You will work across Databricks, Spark, and Snowflake to deliver robust data solutions and ensure data quality and security.

Ideal candidates bring 5+ years in data engineering, strong Python/Scala/Java coding, and hands-on experience with streaming and data transformation tools.