Senior Data Engineer - Lakehouse & Databricks Lead

Agile5 Technologies, Inc.; Washington (District of Columbia); On-site; USD 130,000 - 190,000; Full time

Job summary

Agile5 Technologies, Inc. in Washington, DC is seeking a Senior Data Engineer to lead complex data migration and lakehouse modernization efforts. You will convert legacy ETL to scalable Python/PySpark, migrate to Delta Lake, and implement robust data governance while mentoring junior engineers.

The role requires hands-on Databricks administration, CI/CD pipelines, and collaboration with analysts. This full-time position operates in a secure environment with federal-like responsibilities.

Qualifications

• Minimum experience varies by degree: PhD with 4 years; Master's with 8 years; Bachelor's with 10 years; or High School Diploma with 14 years.
• Hands-on Databricks, Python, PySpark, and SQL for large-scale pipeline development.
• Experience migrating data from Hive/Hadoop/Oracle to cloud-native Delta Lake or Iceberg formats.

Responsibilities

• Lead daily migration operations: data profiling, schema analysis, pipeline conversion, automated reconciliation, and deployment validation.
• Migrate Hive to Delta Lake on AWS S3 using Databricks ingestion tools.
• Convert Informatica mappings to Python/PySpark while preserving business logic and data quality.
• Design CDC pipelines to maintain data synchronization during migration.
• Configure and maintain Databricks in AWS GovCloud.
• Implement Unity Catalog governance with RBAC, encryption, audits and lineage.
• Build automated data reconciliation scripts validating migration accuracy.
• Develop CI/CD pipelines for Python using Azure DevOps and document code.
• Integrate lakehouse with Power BI and ESRI; configure SSO integrations.
• Mentor mid-level engineers; participate in Agile ceremonies; support demos.
• Performs other duties as assigned.

Skills

Databricks

Python

PySpark

SQL

CI/CD

Azure DevOps

Jenkins

GitHub Actions

Power BI

ESRI

Unity Catalog

RBAC

SAML/SSO

Okta

AWS GovCloud

Delta Lake

Hive

Informatica

Education

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related technical field

Tools

Databricks

Informatica

Power BI

ESRI

Azure DevOps

Jenkins

GitHub Actions

Okta

SAML/SSO

AWS GovCloud

Delta Lake

Job description

Agile5 Technologies, Inc. in Washington, DC is seeking a Senior Data Engineer to lead complex data migration and lakehouse modernization efforts. You will convert legacy ETL to scalable Python/PySpark, migrate to Delta Lake, and implement robust data governance while mentoring junior engineers.

The role requires hands-on Databricks administration, CI/CD pipelines, and collaboration with analysts. This full-time position operates in a secure environment with federal-like responsibilities.