Senior Databricks Platform Engineer

General Dynamics IT; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

General Dynamics IT is seeking a Data Platform Engineer to design, implement, and govern a Databricks Lakehouse platform supporting a US Courts modernization program. The role emphasizes scalable data architectures, Unity Catalog governance, and integration with enterprise sources and BI tools.

You will optimize performance, implement security controls, and lead data engineering initiatives across a cloud-based environment, engaging with cross-functional teams and contributing to AI/ML

Qualifications

• 10+ years of related experience in information systems.
• Strong Databricks Lakehouse Platform experience with Unity Catalog and cluster management.
• Experience with CI/CD pipelines and automation tools.
• Proficiency in SQL, Python, and Bash for automation.

Responsibilities

• Design, configure, and maintain the Databricks Lakehouse Platform and Unity Catalog.
• Administer compute resources, clusters, and SQL warehouses for performance.
• Develop data pipelines using Spark, Delta Lake, and Databricks workflows.
• Implement data governance, RBAC, and security controls across the platform.
• Automate provisioning with Terraform and Databricks APIs; monitor performance and cost.
• Collaborate with security, governance, and data teams; mentor engineers.

Skills

Databricks Lakeflow

Databricks Platform

Data Engineering

Data Ingestion

Education

MA/MS degree in information systems or related field

Tools

Docker

Kubernetes

Grafana

Prometheus

Datadog

CloudWatch

Job description

Type of Requisition:

Pipeline

Clearance Level Must Currently Possess:

None

Clearance Level Must Be Able to Obtain:

None

Public Trust/Other Required:

None

Job Family:

Data Science and Data Engineering

Job Qualifications:

• Skills: Databricks Lakeflow, Databricks Platform, Data Engineering, Data Ingestion
• Certifications: None
• Experience: 10 + years of related experience
• US Citizenship Required: No

Job Description:

Seize your opportunity to make a personal impact supporting the Case Management Modernization (CMM) Program. The CMM program is an initiative to support the Administrative Office of the US Courts (AO) in developing a modern cloud-based solution to support all 204+ federal courts across the United States.

GDIT is your place to make meaningful contributions to challenging projects and grow a rewarding career. The Data Platform Engineer will work as part of the CMM Data Modernization and Governance team responsible for delivering an integrated data governance, engineering, data platform, reporting, analytics, and Artificial Intelligence (AI)/Machine Learning (ML) capabilities that support operational decision-making and fulfill AO's data and analytics objectives in support of the CMM program.

The successful candidate will be responsible for designing, building, administrating, optimizing, configuring, maintaining, and governing the organization's Databricks Lakehouse Platform , enabling scalable data engineering, analytics, and governance capabilities in support of the CMM Data Modernization & Governance program.

Responsibilities:

• Design, configure, and maintain the enterprise Databricks Lakehouse Platform, including workspaces, Unity Catalog, and scalable data architectures.
• Administer and optimize Databricks compute, clusters, SQL warehouses, serverless capabilities, and workload management for performance, reliability, and cost efficiency.
• Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
• Create, maintain, and govern Unity Catalog catalogs, schemas, tables, roles/groups, and RBAC/access-control lists, aligned with AO security, IAM, and compliance policies.
• Implement and manage Unity Catalog, RBAC, and data governance controls.
• Automate platform provisioning, configuration, and deployments using Terraform, Python, SQL, Databricks Asset Bundles, and Databricks APIs.
• Implement platform monitoring, logging, alerting, observability, capacity planning, and performance optimization.
• Perform Spark and SQL performance tuning, scalability testing, and troubleshooting of platform, pipeline, and data-processing issues.
• Implement cost optimization strategies, including autoscaling, auto-termination, right-sizing, workload isolation, and contribute to DBU consumption forecasting, financial reporting, and TCO analysis.
• Support platform upgrades, patching, versioning, release management, and adoption of new Databricks capabilities.
• Integrate Databricks with enterprise data sources, governance tools, BI platforms, and AI/ML environments.
• Implement security, encryption, networking, auditing, compliance, and high-availability/disaster-recovery controls.
• Support security audits, ATO activities, incident response, root-cause analysis, and operational reporting.
• Establish platform standards, reusable engineering patterns, reference architectures, runbooks, and technical documentation.
• Collaborate with architecture, security, governance, and data engineering teams and provide technical guidance and mentorship to platform users.
• Maintain cloud monitoring dashboards, capacity planning, and KPI metrics; evaluate new Databricks features and recommend adoption to improve performance, cost, or operability.
• Provide technical guidance and mentorship to data engineers and platform users.

Qualifications:

• MA/MS degree with 12+ years of general experience in information systems and 10+ years of specialized experience.
• Experience may be considered in lieu of degree as follows: HS (18+ years), AA/AS (16+ years), BA/BS (14+ years), Doctorate Degree/Ph.D. (11+ years).
• Strong, hands‑on experience implementing and administering the Databricks Lakehouse Platform, including Unity Catalog, cluster and SQL warehouse management, and job/workflow orchestration.
• Experience in performance engineering, scalability testing, and tuning of Databricks data platform.
• Strong expertise in SQL, Spark performance tuning, and workload management.
• Skilled with performance/observability tools (Grafana, Prometheus, Datadog, CloudWatch, Elastic).
• Hands‑on experience with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, Azure DevOps).
• Proficiency in Python and Bash for automation, testing, and platform scripting; strong SQL skills.
• Experience operating Databricks in AWS‑based or hybrid cloud environments.
• Familiarity with containerized environments (Docker, Kubernetes) and microservices pattern