Onsite Senior Data Engineer - Databricks & Delta Lake Lead

Agile5 Technologies, Inc.; Washington (District of Columbia); On-site; USD 120,000 - 180,000; Full time

Job summary

Agile5 Technologies, Inc. is seeking a Senior Data Engineer in Washington, DC. You will lead migration operations, converting legacy ETL to Python/PySpark and building robust data pipelines in a secure, cloud-native environment.

You will manage Databricks infrastructure, implement CDC, data governance, and CI/CD pipelines, while mentoring junior engineers and collaborating with DBAs and analysts to ensure 100% data accuracy.

Qualifications

• Bachelor's degree required; field as listed above or equivalent combination of education and experience.
• Hands-on experience with Databricks workspace administration, notebooks, and job scheduling.
• Experience migrating data from legacy platforms to cloud-native Delta Lake/Iceberg formats and strong SQL proficiency.

Responsibilities

• Lead data migration and pipeline engineering within enterprise data lakehouse modernization.
• Convert legacy ETL artifacts into Python/PySpark, maintain data quality checks.
• Implement CDC pipelines and data governance with Unity Catalog and RBAC.
• Manage Databricks infrastructure in AWS GovCloud and design CI/CD for Python code.
• Collaborate with analysts and engineers and mentor junior staff.

Skills

Python

PySpark

SQL

Databricks

AWS

CI/CD

Data governance

Delta Lake

ETL/ELT

Education

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related field

Tools

Azure DevOps

Jenkins

GitHub Actions

Job description

Agile5 Technologies, Inc. is seeking a Senior Data Engineer in Washington, DC. You will lead migration operations, converting legacy ETL to Python/PySpark and building robust data pipelines in a secure, cloud-native environment.

You will manage Databricks infrastructure, implement CDC, data governance, and CI/CD pipelines, while mentoring junior engineers and collaborating with DBAs and analysts to ensure 100% data accuracy.