Data Engineer - Delta Lake & Databricks Pipelines

Agile5 Technologies, Inc.; Fairmont (WV); On-site; USD 110,000 - 160,000; Full time

Job summary

Agile5 Technologies, Inc. is seeking a Mid-Level Data Engineer in Washington, DC to support data migration, pipeline modernization, and data lakehouse initiatives.

You will convert Informatica artifacts to Python/PySpark, migrate Hive to Delta Lake on S3, and build automated reconciliation pipelines in a secure federal environment. Using Azure DevOps for version control, you will work with senior engineers to ensure data quality, governance, and robust documentation.

Qualifications

• Proficiency in Python for data transformation and pipeline development, and SQL for query development and schema analysis.
• Experience with ETL/ELT processes, data migration methodologies, and cloud data platforms (AWS, Azure, or GCP).
• Familiarity with version control systems (Azure DevOps, Git) and data quality concepts including profiling, cleansing, and reconciliation.
• Bachelor's degree in Computer Science, Data Engineering, IT, or related field preferred (or equivalent combination of education and experience).

Responsibilities

• Execute daily data migration operations including data profiling, schema mapping, pipeline conversion, and automated reconciliation for Low and Medium complexity Informatica artifacts.
• Convert Informatica mappings into well-documented Python/PySpark code, preserving business logic and data quality controls.
• Migrate legacy Hive tables to Delta Lake format on S3 using Databricks ingestion tools.
• Build and execute automated data reconciliation scripts to validate migration accuracy and establish CDC pipelines.
• Commit converted code into Azure DevOps with documentation and inline comments; maintain Unity Catalog configurations.
• Perform daily data profiling and side-by-side validation within legacy enclave environments.
• Support Power BI and ESRI integration testing and validation.
• Participate in peer code reviews, daily Agile ceremonies, and collaborative data validation sessions.
• Contribute to Data Quality Assessment Reports and support training and knowledge transfer activities.

Skills

Python

PySpark

SQL

ETL/ELT

Data quality

CI/CD

GIT

Education

Bachelor's degree in CS/IT/Data Engineering

Equivalent education/experience

Tools

Databricks

Delta Lake

Apache Spark

Hive

Git

Azure DevOps

Cloud platforms (AWS/Azure/GCP)

Job description

Agile5 Technologies, Inc. is seeking a Mid-Level Data Engineer in Washington, DC to support data migration, pipeline modernization, and data lakehouse initiatives.

You will convert Informatica artifacts to Python/PySpark, migrate Hive to Delta Lake on S3, and build automated reconciliation pipelines in a secure federal environment. Using Azure DevOps for version control, you will work with senior engineers to ensure data quality, governance, and robust documentation.