Databricks Engineer: Scalable ETL/ELT & Data Governance

Jobtailor; Gaithersburg (MD); On-site; USD 120,000 - 180,000; Full time

Job summary

Jobtailor in Gaithersburg, MD seeks a data engineer to design, build, and optimize scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake.

You will develop and maintain Databricks notebooks, jobs, and workflows for high-volume analytics, and collaborate with data scientists, analysts, engineers, and platform teams.

This role emphasizes data governance, cloud security, and CI/CD practices in a regulated environment.

Qualifications

• Bachelor's degree in computer science, software engineering, data engineering, or a related technical field.
• 5+ years of data engineering experience with significant Databricks exposure.
• Strong experience with PySpark, SQL, Spark, Delta Lake, and Databricks.
• Experience building production-grade data pipelines and workflows.
• Experience with cloud data platforms and storage (AWS S3, ADLS, GCS).
• Experience with Git and CI/CD practices.
• Understanding of distributed data processing, data modeling, data quality, and pipeline performance optimization.
• Experience troubleshooting production data workloads.
• Understanding of cloud security concepts like RBAC and data protection.
• Strong communication and collaboration skills.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines with Databricks, PySpark, SQL, and Delta Lake
• Maintain Databricks notebooks, jobs, and workflows for high-volume analytics
• Build data ingestion, transformation, validation, and integration processes
• Migrate and modernize existing data workloads
• Optimize Spark workloads with partitioning, caching, and performance tuning
• Implement automated testing, data quality checks, monitoring, and logging
• Support CI/CD and infrastructure automation with Git, Azure DevOps, or GitHub Actions
• Configure Databricks compute, clusters, runtimes, and job execution
• Enforce RBAC, data governance, and access controls in cloud environments
• Troubleshoot production data issues and improve reliability
• Collaborate across data scientists, analysts, engineers, architects, and platform teams
• Deliver governed datasets for sensitive federal and non-federal data

Skills

Databricks

PySpark

SQL

Apache Spark

Delta Lake

Data Pipeline Development

Data Quality Checks

Performance Tuning

Data Modeling

Infrastructure Automation

Education

Bachelor's degree in computer science, software engineering, data engineering, or a related technical field

Tools

Git

Azure DevOps

GitHub Actions

AWS S3

Azure Data Lake Storage

Google Cloud Storage

Terraform

Databricks Unity Catalog

Databricks APIs

Databricks Lakeflow

Job description

Jobtailor in Gaithersburg, MD seeks a data engineer to design, build, and optimize scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake.

You will develop and maintain Databricks notebooks, jobs, and workflows for high-volume analytics, and collaborate with data scientists, analysts, engineers, and platform teams.

This role emphasizes data governance, cloud security, and CI/CD practices in a regulated environment.