Mid-Level Data Engineer (On-Site in DC)

Agile5 Technologies, Inc.; Fairmont (WV); On-site; USD 110,000 - 160,000; Full time

Job summary

Agile5 Technologies, Inc. is seeking a Mid-Level Data Engineer in Washington, DC to support data migration, pipeline modernization, and data lakehouse initiatives.

You will convert Informatica artifacts to Python/PySpark, migrate Hive to Delta Lake on S3, and build automated reconciliation pipelines in a secure federal environment. Using Azure DevOps for version control, you will work with senior engineers to ensure data quality, governance, and robust documentation.

Qualifications

• Proficiency in Python for data transformation and pipeline development, and SQL for query development and schema analysis.
• Experience with ETL/ELT processes, data migration methodologies, and cloud data platforms (AWS, Azure, or GCP).
• Familiarity with version control systems (Azure DevOps, Git) and data quality concepts including profiling, cleansing, and reconciliation.
• Bachelor's degree in Computer Science, Data Engineering, IT, or related field preferred (or equivalent combination of education and experience).

Responsibilities

• Execute daily data migration operations including data profiling, schema mapping, pipeline conversion, and automated reconciliation for Low and Medium complexity Informatica artifacts.
• Convert Informatica mappings into well-documented Python/PySpark code, preserving business logic and data quality controls.
• Migrate legacy Hive tables to Delta Lake format on S3 using Databricks ingestion tools.
• Build and execute automated data reconciliation scripts to validate migration accuracy and establish CDC pipelines.
• Commit converted code into Azure DevOps with documentation and inline comments; maintain Unity Catalog configurations.
• Perform daily data profiling and side-by-side validation within legacy enclave environments.
• Support Power BI and ESRI integration testing and validation.
• Participate in peer code reviews, daily Agile ceremonies, and collaborative data validation sessions.
• Contribute to Data Quality Assessment Reports and support training and knowledge transfer activities.

Skills

Python

PySpark

SQL

ETL/ELT

Data quality

CI/CD

GIT

Education

Bachelor's degree in CS/IT/Data Engineering

Equivalent education/experience

Tools

Databricks

Delta Lake

Apache Spark

Hive

Git

Azure DevOps

Cloud platforms (AWS/Azure/GCP)

Job description

About Agile5

Agile5 Technologies, Inc., is a Woman-Owned Small Business (WOSB) and Information Technology (IT) services firm that specializes in the design, development, testing, integration, and maintenance of enterprise software systems. We believe our employees are the companys most valuable asset. We are invested in seeing our employees grow in their careers, while maintaining a work/life balance. We have an immediate, full-time need for a skilled, energetic, and driven Mid-Level Data Engineer. Description: The Mid-Level Data Engineer will support data migration, pipeline engineering, and modernization efforts for enterprise data lakehouse architectures. This role involves converting legacy Informatica artifacts into clean Python/PySpark code, migrating database schemas, and building automated data reconciliation pipelines. Working closely with senior engineering leadership and database managers, the ideal candidate will enforce high standards of data quality, data validation, and version control in a secure federal environment.

Job Duties

• Execute daily data migration operations including data profiling, schema mapping, pipeline conversion, and automated reconciliation for Low and Medium complexity Informatica artifacts.
• Convert Informatica mappings into well-documented Python/PySpark code, ensuring all business logic and data quality controls are preserved.
• Migrate legacy Hive tables to Delta Lake format on S3 using Databricks ingestion tools.
• Build and execute automated data reconciliation scripts to validate migration accuracy and establish Change Data Capture (CDC) pipelines for ongoing synchronization.
• Commit all converted code into Azure DevOps with clear documentation and inline comments while maintaining Unity Catalog configurations.
• Perform daily data profiling and side-by-side validation within legacy enclave environments.
• Support Power BI and ESRI integration testing and validation.
• Participate actively in peer code reviews, daily Agile ceremonies, and collaborative data validation sessions.
• Contribute to Data Quality Assessment Reports and support training and knowledge transfer activities.
• Performs other duties as assigned.

Security Clearance Requirements

Public Trust / Tier 4 Eligible: No clearance required to apply; must be a U.S. citizen willing to undergo a background check to obtain a Public Trust / Tier 4 clearance.

Experience Requirements

Minimum experience required varies by degree level: PhD with 0 years; Master's degree with 3 years; Bachelor's degree with 5 years; or High School Diploma with 9 years of relevant experience. Proficiency in Python for data transformation and pipeline development, as well as SQL for query development and schema analysis. Experience with ETL/ELT processes, data migration methodologies, and cloud data platforms (AWS, Azure, or GCP). Familiarity with version control systems (Azure DevOps, Git) and data quality concepts including profiling, cleansing, and reconciliation.

Education Requirements

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field is preferred (or equivalent combination of education and experience).

Desired Skills / Qualifications

• Experience with Databricks (notebooks, jobs, workspace navigation), PySpark, Apache Spark, and Delta Lake or Apache Iceberg table formats.
• Proven track record converting visual ETL tools (Informatica, Talend, SSIS) to code-based pipelines.
• Experience with Hive, HiveQL, or Hadoop ecosystem components.
• Familiarity with federal IT environments, security requirements, and CI/CD pipelines for data engineering workflows.

Location

Washington, DC

Status

Full time

Schedule

Day shift, MondayFriday

Physical Requirements

Must be able to remain in a stationary position for long durations of time. Also, must be able to continuously operate a computer and other office productivity machinery.

Travel Required

No

This Job Description

This job description is subject to change at any time.

Equal Opportunity Employer

W are an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by law.