AWS Databricks Engineer

Credence; McLean (VA); On-site; USD 155,000 - 185,000; Full time

Benefits offered by this job

Health Care Plan

Retirement Plan

Life Insurance

Paid Time Off

Family Leave

Disability Insurance

Training & Development

Wellness Resources

Job summary

Credence is seeking an AWS Databricks Engineer to design, build, and optimize scalable data solutions in the AWS Databricks ecosystem for federal missions. You will implement lakehouse architectures, data pipelines, and governance across Databricks and AWS, collaborating with engineers, data scientists, and stakeholders.

Ideal candidates have hands-on data engineering experience with Databricks, Spark, PySpark, and AWS services, plus a security-conscious mindset and strong communication skills

Qualifications

• U.S. citizenship with the ability to obtain and maintain a Secret clearance is required.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field.
• Hands-on experience in data engineering, cloud engineering, or related technical roles.
• Hands-on experience building and maintaining production data solutions using Databricks, Apache Spark, and PySpark.
• Strong experience with AWS data services such as S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch.
• Strong understanding of ETL/ELT processes, data lake and lakehouse architectures, data modeling, and large-scale data processing.
• Hands-on experience with Delta Lake, SQL, Python, and Spark-based data processing.
• Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows.
• Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline.
• Understanding of cloud security, identity and access management, encryption, and secure data-handling practices.
• Strong communication skills and a client-oriented mindset with the ability to collaborate across technical and non-technical teams.

Responsibilities

• Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake.
• Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture to improve data quality, governance, lineage, performance, and accessibility.
• Build and maintain ETL/ELT pipelines across AWS services such as S3, Glue, RDS, Redshift, Lambda, and related cloud data platforms.
• Develop efficient data transformation workflows using SQL, Python, and PySpark; optimize Spark workloads, clusters, and storage configurations.
• Implement data quality, cataloging, lineage, governance, and access-control practices across Databricks and AWS environments.
• Automate data platform deployments and workflows using IaC, CI/CD pipelines, Git-based development, and AWS deployment tools.
• Collaborate with data engineers, software engineers, data scientists, cloud engineers, and stakeholders to translate requirements into production-ready data solutions.
• Develop clean, well-documented code while following security and federal development standards; support reviews, testing, troubleshooting, and design documentation.

Skills

Databricks engineering

Spark/PySpark

Data pipelines

ETL/ELT

Cloud security

CI/CD

Git/GitHub Actions

Education

Bachelor’s or Master’s degree in CS/Engineering/Data Engineering

Tools

Databricks

Apache Spark

PySpark

Delta Lake

AWS S3

AWS Glue

AWS Redshift

AWS Lambda

IAM

CloudWatch

Terraform

OpenTofu

AWS CDK

CodePipeline

Job description

Overview

Join a team where innovation meets mission. Our AI, cloud, cyber, and modernization solutions save agencies thousands of hours, safeguard national security, and strengthen health and humanitarian missions worldwide. With 1,700+ team members, 1,500+ AI/data experts, and 100+ prime contracts, we deliver at scale and with purpose.

We’ve been recognized as a Top Workplace by the Washington Post for six straight years and named to the Inc. 5000 Fastest Growing Private Companies 13 of the past 14 years. Credence is a welcoming home for those looking to grow and contribute to positive change. We encourage all employees to expand beyond their boundaries and dive into important, world-changing federal challenges.

Position Summary

Credence has an immediate need for an AWS Databricks Engineer to join our growing AI and Automation practice. You will serve as a key technical contributor responsible for designing, building, and optimizing scalable cloud-based data solutions using AWS and Databricks. You’ll develop robust data pipelines, implement modern data lakehouse architectures, and collaborate across engineering, data science, and stakeholder teams to deliver secure, high-performance data capabilities that advance federal missions.

Responsibilities include, but are not limited to, the duties listed below:

• Databricks Engineering
  Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake.
• Data Lakehouse Architecture
  Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture (Bronze/Silver/Gold layers) to improve data quality, governance, lineage, performance, and accessibility.
• AWS Data Integration
  Build and maintain ETL/ELT pipelines integrating structured, semi-structured, and unstructured data across AWS services such as S3, Glue, RDS, Redshift, Lambda, and related cloud data platforms.
• Data Transformation & Optimization
  Develop efficient data transformation workflows using SQL, Python, and PySpark. Optimize Spark workloads, queries, clusters, and storage configurations for performance, reliability, and cost efficiency.
• Data Governance & Quality
  Implement data quality, cataloging, lineage, governance, and access-control practices across Databricks and AWS environments, including the use of Unity Catalog where applicable.
• Cloud Automation & DevOps
  Automate data platform deployments and workflows using Infrastructure as Code (IaC), CI/CD pipelines, Git-based development practices, and AWS-native deployment tools.
• Collaborative Engineering
  Work closely with data engineers, software engineers, data scientists, cloud engineers, and mission stakeholders to translate business and technical requirements into production-ready data solutions.
• Technical Rigor & Documentation
  Develop clean, maintainable, and well-documented code while following engineering, security, and federal development standards. Support code reviews, testing, troubleshooting, and technical design documentation.

• U.S. citizenship with the ability to obtain and maintain a Secret clearance is required.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field.
• Hands‑on experience in data engineering, cloud engineering, or related technical roles.
• Hands‑on experience building and maintaining production data solutions using Databricks, Apache Spark, and PySpark.
• Strong experience with AWS data services such as S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch.
• Strong understanding of ETL/ELT processes, data lake and lakehouse architectures, data modeling, and large‑scale data processing.
• Hands‑on experience with Delta Lake, SQL, Python, and Spark‑based data processing.
• Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows.
• Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline.
• Understanding of cloud security, identity and access management, encryption, and secure data‑handling practices.
• Strong communication skills and a client‑oriented mindset with the ability to collaborate across technical and non‑technical teams.

Preferred

• Experience implementing Databricks Lakehouse Architecture and medallion architecture using Bronze, Silver, and Gold data layers.
• Experience with Databricks Unity Catalog, Delta Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities.
• Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic.
• Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation.
• AWS and/or Databricks certifications, such as AWS Certified Data Engineer - Associate, AWS Certified Solutions Architect, or Databricks Certified Data Engineer credentials.
• Knowledge of federal cybersecurity, RMF, FedRAMP, or other government regulatory and compliance frameworks.
• Experience delivering cloud‑native data solutions within federal government or other highly regulated environments.

Salary Range:$155,000 - $185,000 annually. Actual compensation will be determined based on the selected candidate's experience, education, certifications, skills, and overall qualifications.

• Health Care Plan (Medical, Dental & Vision)
• Retirement Plan (401k, IRA)
• Life Insurance (Basic, Voluntary & AD&D)
• Paid Time Off (Vacation, Sick & Public Holidays)
• Family Leave (Maternity, Paternity)
• Short Term & Long Term Disability
• Training & Development
• Wellness Resources