Remote Databricks Engineer: Scalable Data Pipelines

General Dynamics Corporation; Silver Spring, Northern (MD, KY); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Medical plan options

Paid time off

Disability insurance

Job summary

GDIT is seeking a Databricks Engineer to design, develop, and operate scalable data pipelines for the NIA Data Enclave, enabling researchers to safely work with sensitive datasets. You will collaborate across data science, software, and platform teams to deliver reliable analytics resources.

You will apply Databricks and Spark expertise to ensure data quality, security, scalability, and reliability in mission-focused analytics environments. Strong collaboration and cloud experience are essential.

Qualifications

• Bachelor’s degree in computer science, software engineering, data engineering, or a related technical field.
• 5+ years of data engineering experience, including significant hands-on experience with Databricks.
• Strong experience with PySpark, SQL, Apache Spark, Delta Lake, and Databricks.
• Experience developing production-grade data pipelines and workflows.
• Experience working with cloud-based data platforms and storage such as AWS S3, Azure Data Lake Storage, or Google Cloud Storage.
• Experience with Git and CI/CD practices for deploying and managing data engineering workloads.
• Understanding of distributed data processing, data modeling, data quality, and pipeline performance optimization.
• Experience troubleshooting and supporting production data workloads.
• Understanding of cloud security concepts such as RBAC, identity management, least-privilege access, and data protection.
• Strong communication skills and the ability to collaborate effectively with technical and mission-focused stakeholders.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, and Delta Lake.
• Develop and maintain Databricks notebooks, jobs, and workflows that support high-volume analytical workloads.
• Build reliable data ingestion, transformation, validation, and integration processes.
• Help migrate and modernize existing data workloads for improved scalability, performance, and maintainability.
• Optimize Spark workloads through partitioning, caching, joins, file management, and other performance-tuning techniques.
• Implement automated testing, data quality checks, monitoring, logging, and operational processes.
• Support CI/CD and infrastructure automation for Databricks workloads using Git and tools such as Azure DevOps or GitHub Actions.
• Configure and optimize Databricks compute, clusters, runtimes, and job execution.
• Work within secure, role-based cloud environments and help implement data governance and access controls.
• Troubleshoot production issues, perform root-cause analysis, and improve reliability of data services.
• Collaborate with data scientists, researchers, analysts, and other engineers to deliver high-quality analytical datasets.

Skills

Cloud Technology

Databricks Lakeflow

Databricks Platform

Databricks Unity Catalog

Data Lake

Education

Bachelor's degree in computer science, software engineering, data engineering, or a related technical field

Tools

Databricks

PySpark

SQL

Delta Lake

AWS S3

Azure Data Lake Storage

Job description

GDIT is seeking a Databricks Engineer to design, develop, and operate scalable data pipelines for the NIA Data Enclave, enabling researchers to safely work with sensitive datasets. You will collaborate across data science, software, and platform teams to deliver reliable analytics resources.

You will apply Databricks and Spark expertise to ensure data quality, security, scalability, and reliability in mission-focused analytics environments. Strong collaboration and cloud experience are essential.