Data Engineer (Mid-Level): Lakehouse Migrations & Pipelines

Agile5 Technologies, Inc.; Washington (District of Columbia); On-site; USD 110,000 - 140,000; Full time

Job summary

Agile5 Technologies, Inc. is a Woman-Owned Small Business (WOSB) and IT services firm that specializes in design, development, testing, integration, and maintenance of enterprise software systems.

We hire for growth while balancing work/life in a federal environment. The Mid-Level Data Engineer will support data migration, pipeline engineering, and modernization for data lakehouse architectures, translating Informatica artifacts into Python/PySpark, and ensuring data quality and version control

Qualifications

• Bachelor's degree required or equivalent in CS/IT or related field.
• Experience with data migration and lakehouse architectures in enterprise environments.
• Proficiency translating Informatica artifacts to Python/PySpark code with solid data quality controls.

Responsibilities

• Execute daily data migration workflows including profiling, mapping, and automated reconciliation for low/medium complexity Informatica artifacts.
• Convert Informatica mappings into well-documented Python/PySpark code while preserving business logic and data quality controls.
• Migrate legacy Hive tables to Delta Lake on S3 using Databricks ingestion tools.
• Build and run automated data reconciliation scripts and CDC pipelines for ongoing synchronization.
• Commit converted code to Azure DevOps with documentation and inline comments; maintain Unity Catalog configs.
• Participate in peer code reviews, daily Agile ceremonies, and collaborative data validation sessions.
• Contribute to Data Quality Assessment Reports and support training/knowledge transfer activities.

Skills

Python

SQL

ETL/ELT

PySpark

Apache Spark

Data quality

Informatica knowledge

Hive/Hadoop

CI/CD

Cloud data platforms

Education

Bachelor's degree in CS/IT

Tools

Databricks

Delta Lake

Git

Azure DevOps

Job description

Agile5 Technologies, Inc. is a Woman-Owned Small Business (WOSB) and IT services firm that specializes in design, development, testing, integration, and maintenance of enterprise software systems.

We hire for growth while balancing work/life in a federal environment. The Mid-Level Data Engineer will support data migration, pipeline engineering, and modernization for data lakehouse architectures, translating Informatica artifacts into Python/PySpark, and ensuring data quality and version control