Databricks SME

August Schell; Rockville (MD); On-site; USD 120,000 - 170,000; Full time

Job summary

August Schell is seeking a skilled Data Engineer to lead data management and optimization across batch and streaming workloads. You will architect and maintain ETL pipelines, optimize Databricks/Spark workflows, and ensure data quality and governance in a fast-paced environment.

Ideal candidates have strong Python/Scala/Java, experience with Cribl, Airflow, and real-time processing using Kafka/Flink, plus a BA/BS or higher in a related field and 4+ years of data engineering experience.

Qualifications

• 5+ years’ experience as a Data Engineer with data management and streaming focus.
• Proficiency with Python, Scala and/or Java.
• Expert Databricks and Apache Spark experience mandatory.

Responsibilities

• Data Management Leadership: oversee data at rest and streaming, guide the team.
• ETL Pipeline Development: design and maintain efficient ETL pipelines with data quality.
• Databricks Expertise: optimize processing with Spark-based solutions and integration.
• Streaming Data: manage streaming sources and real-time processing with Kafka/Flink.
• Data Transformation: route, transform, and enrich data with Cribl or similar tools.
• Data Optimization: performance tuning, indexing, compression for storage efficiency.
• Data Security and Compliance: collaborate on privacy and security requirements.
• Documentation: build docs for data processes and pipelines.

Skills

Python

Scala/Java

Databricks

Apache Spark

Kafka

Flink

SQL

NoSQL

Cribl

Airflow

Education

BA/BS in relevant field

MS degree with 2+ years experience

Tools

Databricks

Apache Spark

Airflow

Cribl

Snowflake

Kafka

Flink

Job description

What We Are Looking For…

We are looking for a highly skilled Data Engineer with expert-level knowledge to assume a key role in managing data, both at rest and in streaming, and to lead efforts in data management and optimization. Your proficiency in ETL pipeline development, data tools and your deep understanding of data management principles will be instrumental in ensuring our data infrastructure operates efficiently.

What We Are Looking For…

We are looking for a highly skilled Data Engineer with expert-level knowledge to assume a key role in managing data, both at rest and in streaming, and to lead efforts in data management and optimization. Your proficiency in ETL pipeline development, data tools and your deep understanding of data management principles will be instrumental in ensuring our data infrastructure operates efficiently.

What You Will Do...

• Data Management Leadership: Provide technical leadership in data management, overseeing data at rest and data streaming, and guiding the team
• ETL Pipeline Development: Design, develop, and maintain efficient ETL pipelines to process and transform data, ensuring data quality, integrity, and consistency
• Databricks Expertise: Optimize data processing by using your expertise in Databricks, develop Spark-based solutions, and enhance data integration and analytics capabilities
• Streaming Data: Manage streaming data sources and implement real-time data processing solutions using tools like Apache Kana or similar technologies
• Data Transformation: Implement and manage data routing, transformation, and enrichment using Cribl or similar data pipeline orchestration tools
• Data Optimization: Work on data optimization initiatives, including performance tuning, indexing, and data compression, to ensure efficient data storage and retrieval
• Data Security and Compliance: Collaborate with security teams to ensure data security and compliance with data privacy regulations
• Documentation: Build comprehensive documentation for data management processes, ETL pipelines, and facilitating knowledge transfer within the team

What You Will Bring…

• 5+ years’ experience as a Data Engineer with a deep understanding of data management, ETL pipelines and streaming data
• 5+ years’ experience writing code using Python, Scala and/or Java
• Consultant-level, expert experience in Databricks
• Apache Spark experience (mandatory)
• 2+ years’ expertise in Databricks, Apache Spark, Snowflake and/or similar data processing frameworks
• BA / BS degree with 4+ years of experience (or) MS degree with 2+ years of experience as a data engineer

Stand Out With…

• Experience with data streaming technologies (e.g., Apache Kafka, Apache Flink) and real-time data processing
• Familiarity with AI and machine learning concepts and the ability to work with data scientists and AI engineers
• Experience working on SQL or NoSQL data stores
• Proficiency in tools like Cribl, Airflow or similar for data pipeline orchestration and transformation
• Knowledge of data security and compliance principles

Who We Are...

August Schell offers30 years of experience in providing our customers innovative solutions and engineering services to their most challenging needs. We thrive on navigating complexIT difficulties and are driven to find the best solutions for our customers. Our team delivers expert knowledge in cyber security, big data, endpoint security and cloud solutions. Our ability to serve our customers with continued excellence is a direct reflection of our dedicated and exceptional people. At August Schell, we are not only a team,we are family.

All are welcome…

August Schellis a proud Equal Employment Opportunity and Affiantive Action Employer. We believe that diversity, inclusion, and recognizing that our employees bringunique differences, perspectives, and ideals that are the backbone of oursuccess. We welcome all persons and the talent they have to offer regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, or Veteran status.