Senior Databricks Platform Engineer
General Dynamics Information Technology
Get a reply from this employer
Your resume and cover letter, tailored to exactly what they’re hiring for.
Benefits offered by this job
Job summary
General Dynamics Information Technology, Inc. is seeking a Data Platform Engineer to lead design, configuration, and operation of the Databricks Lakehouse for AO's CMM program.
You will shape data governance, pipelines, and analytics across a cloud-based environment to enable reliable decisions. The role requires hands-on expertise with Unity Catalog, RBAC, Terraform, Python, SQL, and Spark tuning, plus collaboration with security and governance teams.
Qualifications
- Extensive experience implementing and administering the Databricks Lakehouse Platform.
- Experience with performance engineering, scalability testing, and tuning of Databricks data platform.
- Strong expertise in SQL, Spark tuning, and workload management.
- Hands-on experience with CI/CD pipelines and IaC automation.
Responsibilities
- Design, configure, and maintain the enterprise Databricks Lakehouse Platform.
- Administer and optimize Databricks compute, clusters, SQL warehouses, serverless capabilities, and workload management.
- Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Spark, Delta Lake, and Databricks Workflows.
- Create, maintain, and govern Unity Catalog catalogs, schemas, tables, roles/groups, and RBAC.
- Implement and manage Unity Catalog, RBAC, and data governance controls.
- Automate platform provisioning, configuration, and deployments using Terraform, Python, SQL, and Databricks APIs.
- Implement platform monitoring, logging, observability, and cost optimization.
- Perform Spark and SQL performance tuning, scalability testing, and trouble-shooting.
- Collaborate with architecture, security, governance, and data engineering teams; mentor platform users.
Skills
Education
Tools
Job description
Seize your opportunity to make a personal impact supporting the Case Management Modernization (CMM) Program. The CMM program is an initiative to support the Administrative Office of the US Courts (AO) in developing a modern cloud-based solution to support all 204+ federal courts across the United States. GDIT is your place to make meaningful contributions to challenging projects and grow a rewarding career. The Data Platform Engineer will work as part of the CMM Data Modernization and Governance team responsible for delivering an integrated data governance, engineering, data platform, reporting, analytics, and Artificial Intelligence (AI)/Machine Learning (ML) capabilities that support operational decision-making and fulfill AO's data and analytics objectives in support of the CMM program.
The Data Platform Engineer will execute the following responsibilities:
- Design, configure, and maintain the enterprise Databricks Lakehouse Platform, including workspaces, Unity Catalog, and scalable data architectures.
- Administer and optimize Databricks compute, clusters, SQL warehouses, serverless capabilities, and workload management for performance, reliability, and cost efficiency.
- Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
- Create, maintain, and govern Unity Catalog catalogs, schemas, tables, roles/groups, and RBAC/access-control lists, aligned with AO security, IAM, and compliance policies.
- Implement and manage Unity Catalog, RBAC, and data governance controls.
- Automate platform provisioning, configuration, and deployments using Terraform, Python, SQL, Databricks Asset Bundles, and Databricks APIs.
- Implement platform monitoring, logging, alerting, observability, capacity planning, and performance optimization.
- Perform Spark and SQL performance tuning, scalability testing, and troubleshooting of platform, pipeline, and data-processing issues.
- Implement cost optimization strategies, including autoscaling, auto-termination, right-sizing, workload isolation, and contribute to DBU consumption forecasting, financial reporting, and TCO analysis.
- Support platform upgrades, patching, versioning, release management, and adoption of new Databricks capabilities.
- Integrate Databricks with enterprise data sources, governance tools, BI platforms, and AI/ML environments.
- Implement security, encryption, networking, auditing, compliance, and high-availability/disaster-recovery controls.
- Support security audits, ATO activities, incident response, root-cause analysis, and operational reporting.
- Establish platform standards, reusable engineering patterns, reference architectures, runbooks, and technical documentation.
- Collaborate with architecture, security, governance, and data engineering teams and provide technical guidance and mentorship to platform users.
- Maintain cloud monitoring dashboards, capacity planning, and KPI metrics; evaluate new Databricks features and recommend adoption to improve performance, cost, or operability.
- Provide technical guidance and mentorship to data engineers and platform users.
- MA/MS degree with 12+ years of general experience in information systems and 10+ years of specialized experience.
- Experience may be considered in lieu of degree as follows: HS (18+ years), AA/AS (16+ years), BA/BS (14+ years), Doctorate Degree/Ph.D. (11+ years).
- Strong, hands-on experience implementing and administering the Databricks Lakehouse Platform, including Unity Catalog, cluster and SQL warehouse management, and job/workflow orchestration.
- Experience in performance engineering, scalability testing, and tuning of Databricks data platform.
- Strong expertise in SQL, Spark performance tuning, and workload management.
- Skilled with performance/observability tools (Grafana, Prometheus, Datadog, CloudWatch, Elastic).
- Hands-on experience with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, Azure DevOps).
- Proficiency in Python and Bash for automation, testing, and platform scripting; strong SQL skills.
- Experience operating Databricks in AWS-based or hybrid cloud environments.
- Familiarity with containerized environments (Docker, Kubernetes) and microservices patterns.
- Experience with telemetry, logging standards, traceability, and root-cause analysis.
- Ability to analyze large performance datasets to identify trends, issues, and optimization opportunities.
- Experience with version control, build/release processes, and IaC automation.
- Knowledge of data ingestion, ETL/ELT, lake/warehouse architectures, and distributed data processing.
- Understanding of cloud security, IAM, access control, networking, and resource governance within Databricks and Unity Catalog.
- Familiarity with federal security, compliance, and audit processes (ATO, FedRAMP).
- Strong troubleshooting skills and ability to optimize workload performance.
- Databricks Certified Data Engineer Associate or Professional
- Databricks Certified Associate Platform Administrator
- Databricks Certified Machine Learning Professional
- Databricks Certified Associate Developer for Apache Spark
- Certified Data Management Professional (CDMP)
- AWS Certified Data Analytics - Specialty
- AWS Certified Solutions Architect - Professional
- AWS Certified Data Engineer
- Docker Certified Associate
- Certified Professional DevSecOps Certification Course
- ITIL, AWS SysOps
The likely salary range for this position is $186,542 - $252,380. This is not, however, a guarantee of compensation or salary. Rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range.
40
None
Remote
Any Location / Remote
No
Our benefits package for all US-based employees includes a variety of medical plan options, some with Health Savings Accounts, dental plan options, a vision plan, and a 401(k) plan offering the ability to contribute both pre and post-tax dollars up to the IRS annual limits and receive a company match. To encourage work/life balance, GDIT offers employees full flex work weeks where possible and a variety of paid time off plans, including vacation, sick and personal time, holidays, paid parental, military, bereavement and jury duty leave. GDIT typically provides new employees with 15 days of paid leave per calendar year to be used for vacations, personal business, and illness and an additional 10 paid holidays per year. Paid leave and paid holidays are prorated based on the employee's date of hire. The GDIT Paid Family Leave program provides a total of up to 160 hours of paid leave in a rolling 12 month period for eligible employees. To ensure our employees are able to protect their income, other offerings such as short and long-term disability benefits, life, accidental death and dismemberment, personal accident, critical illness and business travel and accident insurance are provided or available. We regularly review our Total Rewards package to ensure our offerings are competitive and reflect what our employees have told us they value most.
As part of the hiring process, we will ask you to complete an identity verification process that leverages advanced biometrics and artificial intelligence to ensure authenticity and protect against identity fraud. You are expected to be on camera during virtual interviews. We reserve the right to take your picture to verify your identity and prevent fraud. By proceeding, you authorize the collection, processing, and use of your biometric data for identity verification and security purposes.
We are GDIT. A global technology and professional services company that delivers technology solutions and mission services to every major agency across the U.S. government, defense and intelligence community. Our 26,000 experts extract the power of technology to create immediate value and deliver solutions at the edge of innovation. We operate across 50+ countries worldwide, offering leading mission-ready capabilities in AI, cloud, cyber and software development.
Equal Opportunity Employer / Individuals with Disabilities / Protected Veterans
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
