Databricks Engineer: Build Secure, Scalable Data Pipelines

General Dynamics Information Technology; Gaithersburg (MD); On-site; USD 120,000 - 180,000; Full time

Job summary

General Dynamics Information Technology seeks a Databricks Engineer to design, build, and operate scalable data pipelines for the NIA Data Enclave, enabling researchers to securely access sensitive datasets.

You will partner with data scientists, engineers, and platform teams to optimize Spark workloads, implement data governance, and automate CI/CD for Databricks workloads in a secure RBAC-enabled cloud environment.

Qualifications

• Bachelor’s degree in a related technical field.
• 5+ years of data engineering experience with Databricks.
• Strong experience with PySpark, SQL, Spark, Delta Lake, and Databricks.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, and Delta Lake.
• Develop and maintain Databricks notebooks, jobs, and workflows that support high-volume analytical workloads.
• Build data ingestion, transformation, validation, and integration processes.
• Migrate and modernize existing data workloads for scalability, performance, and maintainability.
• Optimize Spark workloads with partitioning, caching, joins, and file management.
• Implement automated testing, data quality checks, monitoring, logging, and operational processes.
• Support CI/CD and infrastructure automation for Databricks workloads using Git and Azure DevOps or GitHub Actions.
• Configure and optimize Databricks compute, clusters, runtimes, and job execution.
• Work in secure cloud environments with data governance and access controls.
• Troubleshoot production issues and perform root-cause analysis.
• Collaborate with data scientists, researchers, analysts, and engineers to deliver datasets.

Skills

Data engineering

Cloud collaboration

Communication

Education

Bachelor's degree in CS/SE/Data Eng

Tools

Databricks

PySpark

SQL

Apache Spark

Delta Lake

AWS S3 / Azure Data Lake / GCS

Git

CI/CD

Terraform

Job description

General Dynamics Information Technology seeks a Databricks Engineer to design, build, and operate scalable data pipelines for the NIA Data Enclave, enabling researchers to securely access sensitive datasets.

You will partner with data scientists, engineers, and platform teams to optimize Spark workloads, implement data governance, and automate CI/CD for Databricks workloads in a secure RBAC-enabled cloud environment.