Databricks Data Engineering Lead - ETL, Spark & Streaming

OpenTalent; Rockville (MD); On-site; USD 130,000 - 180,000; Full time

Job summary

OpenTalent is seeking a senior Data Engineer to lead data management initiatives, design robust ETL pipelines, and optimize Databricks/Spark-based processing. You will guide a team in handling data at rest and streaming, ensuring quality, integrity, and timely insights.

The role requires deep Databricks expertise, Python/Scala/Java coding skills, and experience with Kafka/Flink, Snowflake, Cribl, and Airflow. Collaboration with data scientists and security/compliance is essential.

Qualifications

• 5+ years’ experience as a Data Engineer with data management and streaming.
• 5+ years’ experience writing code in Python, Scala and/or Java.
• Consultant-level, expert experience in Databricks.
• Apache Spark experience mandatory.
• 2+ years’ expertise in Databricks, Spark, Snowflake or similar data processing frameworks.
• BA/BS with 4+ years experience or MS with 2+ years as a data engineer.
• Experience with data streaming technologies (Kafka, Flink) and real-time processing.
• Experience with SQL or NoSQL data stores.
• Proficiency with Cribl, Airflow or similar for orchestration.
• Knowledge of data security and compliance principles.
• Familiarity with AI/ML concepts and collaboration with data scientists.

Responsibilities

• Data management leadership, guiding data at rest and streaming initiatives.
• Design, develop, and maintain ETL pipelines for data processing and quality.
• Optimize data processing in Databricks and Spark-based solutions.
• Manage streaming data sources and real-time processing implementations.
• Implement routing, transformation, and enrichment using Cribl or similar tools.
• Drive data optimization, performance tuning, and storage efficiency.
• Collaborate with security teams to ensure data privacy compliance.
• Create comprehensive documentation for data pipelines and processes.
• Mentor team members and facilitate knowledge transfer.

Skills

Databricks

Apache Spark

Python/Scala/Java

SQL/NoSQL

Kafka/Flink

Airflow/Cribl

Data security

Documentation

Education

BA/BS in CS/IS

MS in CS/SE

Tools

Snowflake

Job description

OpenTalent is seeking a senior Data Engineer to lead data management initiatives, design robust ETL pipelines, and optimize Databricks/Spark-based processing. You will guide a team in handling data at rest and streaming, ensuring quality, integrity, and timely insights.

The role requires deep Databricks expertise, Python/Scala/Java coding skills, and experience with Kafka/Flink, Snowflake, Cribl, and Airflow. Collaboration with data scientists and security/compliance is essential.