Senior Data Engineer (On Site, Washington, DC) with Security Clearance

Agile5 Technologies, Inc.; Washington (District of Columbia); On-site; USD 130,000 - 190,000; Full time

Job summary

Agile5 Technologies, Inc. in Washington, DC is seeking a Senior Data Engineer to lead complex data migration and lakehouse modernization efforts. You will convert legacy ETL to scalable Python/PySpark, migrate to Delta Lake, and implement robust data governance while mentoring junior engineers.

The role requires hands-on Databricks administration, CI/CD pipelines, and collaboration with analysts. This full-time position operates in a secure environment with federal-like responsibilities.

Qualifications

• Minimum experience varies by degree: PhD with 4 years; Master's with 8 years; Bachelor's with 10 years; or High School Diploma with 14 years.
• Hands-on Databricks, Python, PySpark, and SQL for large-scale pipeline development.
• Experience migrating data from Hive/Hadoop/Oracle to cloud-native Delta Lake or Iceberg formats.

Responsibilities

• Lead daily migration operations: data profiling, schema analysis, pipeline conversion, automated reconciliation, and deployment validation.
• Migrate Hive to Delta Lake on AWS S3 using Databricks ingestion tools.
• Convert Informatica mappings to Python/PySpark while preserving business logic and data quality.
• Design CDC pipelines to maintain data synchronization during migration.
• Configure and maintain Databricks in AWS GovCloud.
• Implement Unity Catalog governance with RBAC, encryption, audits and lineage.
• Build automated data reconciliation scripts validating migration accuracy.
• Develop CI/CD pipelines for Python using Azure DevOps and document code.
• Integrate lakehouse with Power BI and ESRI; configure SSO integrations.
• Mentor mid-level engineers; participate in Agile ceremonies; support demos.
• Performs other duties as assigned.

Skills

Databricks

Python

PySpark

SQL

CI/CD

Azure DevOps

Jenkins

GitHub Actions

Power BI

ESRI

Unity Catalog

RBAC

SAML/SSO

Okta

AWS GovCloud

Delta Lake

Hive

Informatica

Education

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related technical field

Tools

Databricks

Informatica

Power BI

ESRI

Azure DevOps

Jenkins

GitHub Actions

Okta

SAML/SSO

AWS GovCloud

Delta Lake

Job description

About Agile5: Agile5 Technologies, Inc., is a Woman-Owned Small Business (WOSB) and Information Technology (IT) services firm that specializes in the design, development, testing, integration, and maintenance of enterprise software systems. We believe our employees are the company's most valuable asset. We are invested in seeing our employees grow in their careers, while maintaining a work/life balance. We have an immediate, full-time need for a skilled, energetic, and driven Senior Data Engineer. Description: The Senior Data Engineer serves as the technical lead for complex data migration and pipeline engineering initiatives within enterprise data lakehouse modernization efforts. This role leads the conversion of legacy ETL artifacts into scalable Python/PySpark code, migrates database systems to cloud-native Delta Lake architectures, and implements robust data governance and automated reconciliation frameworks. Operating in a secure environment, the ideal candidate will manage Databricks infrastructure, design CI/CD pipelines, and mentor engineering personnel while ensuring complete data accuracy.

Senior Data Engineer Job Duties:

• Serve as the primary technical lead executing daily migration operations, including data profiling, schema analysis, pipeline conversion, automated reconciliation, and production deployment validation.
• Migrate legacy Hive tables to Delta Lake format on AWS S3 storage using Databricks ingestion tools.
• Convert High and Medium complexity Informatica mappings to Python/PySpark code while preserving all business logic, data quality checks, and data cleansing routines.
• Design and implement Change Data Capture (CDC) pipelines to maintain data synchronization between legacy and target environments during migration.
• Configure and maintain Databricks workspaces, clusters, notebooks, and jobs within AWS GovCloud environments.
• Implement Unity Catalog data governance, including RBAC, column-level encryption, audit trails, and data lineage tracking.
• Build and execute automated data reconciliation scripts validating 100% migration accuracy.
• Develop and maintain CI/CD pipelines for Python code deployment using Azure DevOps, checking in all converted code with comprehensive documentation.
• Integrate the Databricks lakehouse with downstream applications (such as Power BI and ESRI) and configure SSO integrations.
• Access legacy enclave environments for data profiling and side-by-side validation while collaborating daily with database managers and analysts.
• Mentor mid-level engineering staff on platform operations and migration methodologies, participate in Agile ceremonies, and support system acceptance demonstrations.
• Performs other duties as assigned.

Security Clearance Requirements: Public Trust / Tier 4 Eligible: No clearance required to apply; must be a U.S. citizen willing to undergo a background check to obtain a Public Trust / Tier 4 clearance.

Experience Requirements: Minimum experience required varies by degree level: PhD with 4 years; Master's degree with 8 years; Bachelor's degree with 10 years; or High School Diploma with 14 years of relevant experience.

• Hands-on experience with Databricks (workspace administration, notebook development, job scheduling) and proficiency in Python, PySpark, and SQL for large-scale pipeline development.
• Experience migrating data from legacy platforms (Hive, Hadoop, Oracle) to cloud-native platforms, utilizing Delta Lake or Apache Iceberg table formats.
• Practical experience with AWS services (S3, IAM, RDS, GovCloud), CI/CD pipeline tools (Azure DevOps, Jenkins, GitHub Actions), and enterprise ETL/ELT frameworks.

Education Requirements: Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related technical field (or equivalent combination of education and experience).

Desired Skills / Qualifications: Databricks Certified Data Engineer (Associate or Professional).

• Direct experience converting Informatica PowerCenter mappings to Python/PySpark code.
• Experience with Unity Catalog data governance, Cloudera Hive/HiveQL, Power BI/ESRI integrations, and CDC methodologies.
• Familiarity with FISMA High or FedRAMP High security requirements and SAML-based SSO/Okta integrations.

Location: Washington, DC Status: Full time Schedule: Day shift, Monday-Friday Physical Requirements: Must be able to remain in a stationary position for long durations of time. Also, must be able to continuously operate a computer and other office productivity machinery. Travel Required: No This job description is subject to change at any time.

We are an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by law.