Databricks SME - August Schell

OpenTalent; Rockville (MD); On-site; USD 130,000 - 180,000; Full time

Job summary

OpenTalent is seeking a senior Data Engineer to lead data management initiatives, design robust ETL pipelines, and optimize Databricks/Spark-based processing. You will guide a team in handling data at rest and streaming, ensuring quality, integrity, and timely insights.

The role requires deep Databricks expertise, Python/Scala/Java coding skills, and experience with Kafka/Flink, Snowflake, Cribl, and Airflow. Collaboration with data scientists and security/compliance is essential.

Qualifications

• 5+ years’ experience as a Data Engineer with data management and streaming.
• 5+ years’ experience writing code in Python, Scala and/or Java.
• Consultant-level, expert experience in Databricks.
• Apache Spark experience mandatory.
• 2+ years’ expertise in Databricks, Spark, Snowflake or similar data processing frameworks.
• BA/BS with 4+ years experience or MS with 2+ years as a data engineer.
• Experience with data streaming technologies (Kafka, Flink) and real-time processing.
• Experience with SQL or NoSQL data stores.
• Proficiency with Cribl, Airflow or similar for orchestration.
• Knowledge of data security and compliance principles.
• Familiarity with AI/ML concepts and collaboration with data scientists.

Responsibilities

• Data management leadership, guiding data at rest and streaming initiatives.
• Design, develop, and maintain ETL pipelines for data processing and quality.
• Optimize data processing in Databricks and Spark-based solutions.
• Manage streaming data sources and real-time processing implementations.
• Implement routing, transformation, and enrichment using Cribl or similar tools.
• Drive data optimization, performance tuning, and storage efficiency.
• Collaborate with security teams to ensure data privacy compliance.
• Create comprehensive documentation for data pipelines and processes.
• Mentor team members and facilitate knowledge transfer.

Skills

Databricks

Apache Spark

Python/Scala/Java

SQL/NoSQL

Kafka/Flink

Airflow/Cribl

Data security

Documentation

Education

BA/BS in CS/IS

MS in CS/SE

Tools

Snowflake

Job description

• Data Management Leadership: Provide technical leadership in data management, overseeing data at rest and data streaming, and guiding the team.
• ETL Pipeline Development: Design, develop, and maintain efficient ETL pipelines to process and transform data, ensuring data quality, integrity, and consistency.
• Databricks Expertise: Optimize data processing by using your expertise in Databricks, develop Spark-based solutions, and enhance data integration and analytics capabilities.
• Streaming Data: Manage streaming data sources and implement real-Ime data processing solutions using tools like Apache Kana or similar technologies.
• Data Transformation: Implement and manage data routing, transformation, and enrichment using Cribl or similar data pipeline orchestration tools.
• Data Optimization: Work on data optimization initiatives, including performance tuning, indexing, and data compression, to ensure efficient data storage and retrieval.
• Data Security and Compliance: Collaborate with security teams to ensure data security and compliance with data privacy regulations.
• Documentation: Build comprehensive documentation for data management processes, ETL pipelines, and facilitating knowledge transfer within the team.
• 5+ years’ experience as a Data Engineer with a deep understanding of data management, ETL pipelines and streaming data.
• 5+ years’ experience writing code using Python, Scala and/or Java.
• Consultant-level, expert experience in Databricks.
• Apache Spark experience (mandatory).
• 2+ years’ expertise in Databricks, Apache Spark, Snowflake and/or similar data processing frameworks.
• BA / BS degree with 4+ years of experience (or) MS degree with 2+ years of experience as a data engineer.
• Experience with data streaming technologies (e.g., Apache Kafka, Apache Flink) and real-time data processing.
• Familiarity with AI and machine learning concepts and the ability to work with data scientists and AI engineers.
• Experience working on SQL or NoSQL data stores.
• Proficiency in tools like Cribl, Airflow or similar for data pipeline orchestration and transformation.
• Knowledge of data security and compliance principles.

Wl1yiNGi7Z