Data Engineer, PySpark & Delta Lake (On-Site in DC)

Agile5 Technologies, Inc.; Washington (Washington County); Hybrid; USD 68,000 - 152,000; Full time

Job summary

Agile5 Technologies, Inc. in Washington, DC is seeking a Mid-Level Data Engineer to work on-site and support data migration, pipeline modernization, and enterprise lakehouse initiatives.

You will convert legacy Informatica artifacts to Python/PySpark, migrate schemas, and build automated reconciliation pipelines in a secure federal environment. Work closely with senior engineers and database managers, enforcing data quality and version control while contributing to CI/CD, Delta Lake, and

Qualifications

• Proficiency in Python for data transformation and pipeline development.
• Strong SQL skills for query development and schema analysis.
• Experience with ETL/ELT and data migration methodologies.
• Familiarity with cloud data platforms (AWS, Azure, or GCP).

Responsibilities

• Execute daily data migration operations and profile data.
• Convert Informatica mappings to Python/PySpark with preserved business logic.
• Migrate legacy Hive tables to Delta Lake on S3 using Databricks tools.
• Build automated reconciliation scripts and CDC pipelines for synchronization.
• Collaborate in Agile ceremonies and peer code reviews.

Skills

Python

SQL

Data migration

ETL/ELT

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Azure DevOps

Git

Delta Lake

Job description

Agile5 Technologies, Inc. in Washington, DC is seeking a Mid-Level Data Engineer to work on-site and support data migration, pipeline modernization, and enterprise lakehouse initiatives.

You will convert legacy Informatica artifacts to Python/PySpark, migrate schemas, and build automated reconciliation pipelines in a secure federal environment. Work closely with senior engineers and database managers, enforcing data quality and version control while contributing to CI/CD, Delta Lake, and