Mid-Level Data Engineer (On-Site in Washington, DC) with Security Clearance

Agile5 Technologies, Inc.; Washington (District of Columbia); On-site; USD 110,000 - 140,000; Full time

Job summary

Agile5 Technologies, Inc. is a Woman-Owned Small Business (WOSB) and IT services firm that specializes in design, development, testing, integration, and maintenance of enterprise software systems.

We hire for growth while balancing work/life in a federal environment. The Mid-Level Data Engineer will support data migration, pipeline engineering, and modernization for data lakehouse architectures, translating Informatica artifacts into Python/PySpark, and ensuring data quality and version control

Qualifications

• Bachelor's degree required or equivalent in CS/IT or related field.
• Experience with data migration and lakehouse architectures in enterprise environments.
• Proficiency translating Informatica artifacts to Python/PySpark code with solid data quality controls.

Responsibilities

• Execute daily data migration workflows including profiling, mapping, and automated reconciliation for low/medium complexity Informatica artifacts.
• Convert Informatica mappings into well-documented Python/PySpark code while preserving business logic and data quality controls.
• Migrate legacy Hive tables to Delta Lake on S3 using Databricks ingestion tools.
• Build and run automated data reconciliation scripts and CDC pipelines for ongoing synchronization.
• Commit converted code to Azure DevOps with documentation and inline comments; maintain Unity Catalog configs.
• Participate in peer code reviews, daily Agile ceremonies, and collaborative data validation sessions.
• Contribute to Data Quality Assessment Reports and support training/knowledge transfer activities.

Skills

Python

SQL

ETL/ELT

PySpark

Apache Spark

Data quality

Informatica knowledge

Hive/Hadoop

CI/CD

Cloud data platforms

Education

Bachelor's degree in CS/IT

Tools

Databricks

Delta Lake

Git

Azure DevOps

Job description

About Agile5: Agile5 Technologies, Inc., is a Woman-Owned Small Business (WOSB) and Information Technology (IT) services firm that specializes in the design, development, testing, integration, and maintenance of enterprise software systems. We believe our employees are the company's most valuable asset. We are invested in seeing our employees grow in their careers, while maintaining a work/life balance. We have an immediate, full-time need for a skilled, energetic, and driven Mid-Level Data Engineer. Description: The Mid-Level Data Engineer will support data migration, pipeline engineering, and modernization efforts for enterprise data lakehouse architectures. This role involves converting legacy Informatica artifacts into clean Python/PySpark code, migrating database schemas, and building automated data reconciliation pipelines. Working closely with senior engineering leadership and database managers, the ideal candidate will enforce high standards of data quality, data validation, and version control in a secure federal environment.

Mid-Level Data Engineer Job Duties

* Execute daily data migration operations including data profiling, schema mapping, pipeline conversion, and automated reconciliation for Low and Medium complexity Informatica artifacts.

• Convert Informatica mappings into well-documented Python/PySpark code, ensuring all business logic and data quality controls are preserved.
• Migrate legacy Hive tables to Delta Lake format on S3 using Databricks ingestion tools.
• Build and execute automated data reconciliation scripts to validate migration accuracy and establish Change Data Capture (CDC) pipelines for ongoing synchronization.
• Commit all converted code into Azure DevOps with clear documentation and inline comments while maintaining Unity Catalog configurations.
• Perform daily data profiling and side-by-side validation within legacy enclave environments.
• Support Power BI and ESRI integration testing and validation.
• Participate actively in peer code reviews, daily Agile ceremonies, and collaborative data validation sessions.
• Contribute to Data Quality Assessment Reports and support training and knowledge transfer activities.
• Performs other duties as assigned.

Security Clearance Requirements

* Public Trust / Tier 4 Eligible: No clearance required to apply; must be a U.S. citizen willing to undergo a background check to obtain a Public Trust / Tier 4 clearance.

Experience Requirements

* Minimum experience required varies by degree level: PhD with 0 years; Master's degree with 3 years; Bachelor's degree with 5 years; or High School Diploma with 9 years of relevant experience.

• Proficiency in Python for data transformation and pipeline development, as well as SQL for query development and schema analysis.
• Experience with ETL/ELT processes, data migration methodologies, and cloud data platforms (AWS, Azure, or GCP).
• Familiarity with version control systems (Azure DevOps, Git) and data quality concepts including profiling, cleansing, and reconciliation.

Education Requirements

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field is preferred (or equivalent combination of education and experience).

Desired Skills / Qualifications

* Experience with Databricks (notebooks, jobs, workspace navigation), PySpark, Apache Spark, and Delta Lake or Apache Iceberg table formats.

• Proven track record converting visual ETL tools (Informatica, Talend, SSIS) to code-based pipelines.
• Experience with Hive, HiveQL, or Hadoop ecosystem components.
• Familiarity with federal IT environments, security requirements, and CI/CD pipelines for data engineering workflows.

Location: Washington, DC Status: Full time Schedule: Day shift, Monday-Friday Physical Requirements: Must be able to remain in a stationary position for long durations of time. Also, must be able to continuously operate a computer and other office productivity machinery. Travel Required: No

We are an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by law.