Mid-Level Data Engineer: PySpark, Delta Lake, Databricks

Agile5 Technologies, Inc.

Washington (District of Columbia)

On-site

USD 100,000 - 140,000

Full time

2 days ago

Be an early applicant

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Agile5 Technologies, Inc. seeks a Mid-Level Data Engineer to support data migration, pipeline engineering, and modernization for enterprise lakehouse architectures in a federal setting. You will convert legacy Informatica artifacts to Python/PySpark and build automated reconciliation pipelines.

You will work with senior leadership to ensure data quality, version control, and secure data handling while collaborating in daily Agile ceremonies.

Qualifications

Proficiency in Python for data transformation and pipeline development.
Strong SQL skills for query development and schema analysis.
Experience with ETL/ELT, data migration, and cloud data platforms (AWS/Azure/GCP).

Responsibilities

Execute daily data migration operations including profiling, mapping, and pipeline conversion.
Convert Informatica artifacts to Python/PySpark with preserved business logic.
Migrate Hive tables to Delta Lake format on S3 using Databricks ingestion tools.
Build automated data reconciliation scripts and CDC pipelines for sync.
Commit code into Azure DevOps with documentation and Unity Catalog configs.
Support data validation, testing with Power BI/ESRI, and peer reviews.
Participate in Agile ceremonies and knowledge transfer activities.
Contribute to Data Quality Assessment Reports.