Senior Data Engineer (On Site in DC)

Agile5 Technologies, Inc.; Fairmont (WV); On-site; USD 140,000 - 200,000; Full time

Job summary

Agile5 Technologies, Inc. is seeking a Senior Data Engineer to lead complex migration projects and build scalable data pipelines in a secure AWS GovCloud Databricks environment.

You will convert legacy ETL artifacts to Python/PySpark, migrate Hive to Delta Lake, implement CDC pipelines, and govern data with Unity Catalog while mentoring engineers and collaborating across teams. A strong background in Python, PySpark, SQL, and CI/CD is required.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, IT, or a related technical field.
• Hands-on experience with Databricks (workspace administration, notebook development, job scheduling) and proficiency in Python, PySpark, and SQL.
• Experience migrating data from legacy platforms (Hive, Hadoop, Oracle) to cloud-native platforms (Delta Lake/Athena).
• Experience with AWS services (S3, IAM, GovCloud) and CI/CD tools (Azure DevOps, Jenkins, GitHub Actions).

Responsibilities

• Lead daily migration operations, data profiling, schema analysis, and deployment validation.
• Migrate Hive tables to Delta Lake format on AWS S3 using Databricks tools.
• Translate Informatica mappings to Python/PySpark preserving business logic and data quality checks.
• Design CDC pipelines to synchronize data during migration.
• Configure Databricks workspaces, clusters, notebooks, and jobs in AWS GovCloud.
• Implement Unity Catalog governance, RBAC, encryption, audit trails, and data lineage.
• Build and run automated data reconciliation scripts to ensure migration accuracy.
• Develop CI/CD pipelines for Python deployment using Azure DevOps and related tooling.
• Mentor engineers, participate in Agile ceremonies, and support demonstrations.

Skills

Python

PySpark

SQL

Data migration

Leadership

Education

Bachelor's degree in Computer Science, Data Engineering, IT, or related field

Tools

Databricks

AWS GovCloud

Azure DevOps

Jenkins

GitHub Actions

Power BI

ESRI

Job description

About Agile5: Agile5 Technologies, Inc., is a Woman-Owned Small Business (WOSB) and Information Technology (IT) services firm that specializes in the design, development, testing, integration, and maintenance of enterprise software systems. We believe our employees are the companys most valuable asset. We are invested in seeing our employees grow in their careers, while maintaining a work/life balance. We have an immediate, full-time need for a skilled, energetic, and driven Senior Data Engineer. Description: The Senior Data Engineer serves as the technical lead for complex data migration and pipeline engineering initiatives within enterprise data lakehouse modernization efforts. This role leads the conversion of legacy ETL artifacts into scalable Python/PySpark code, migrates database systems to cloud-native Delta Lake architectures, and implements robust data governance and automated reconciliation frameworks. Operating in a secure environment, the ideal candidate will manage Databricks infrastructure, design CI/CD pipelines, and mentor engineering personnel while ensuring complete data accuracy. Senior Data Engineer Job Duties: Serve as the primary technical lead executing daily migration operations, including data profiling, schema analysis, pipeline conversion, automated reconciliation, and production deployment validation. Migrate legacy Hive tables to Delta Lake format on AWS S3 storage using Databricks ingestion tools. Convert High and Medium complexity Informatica mappings to Python/PySpark code while preserving all business logic, data quality checks, and data cleansing routines. Design and implement Change Data Capture (CDC) pipelines to maintain data synchronization between legacy and target environments during migration. Configure and maintain Databricks workspaces, clusters, notebooks, and jobs within AWS GovCloud environments. Implement Unity Catalog data governance, including RBAC, column-level encryption, audit trails, and data lineage tracking. Build and execute automated data reconciliation scripts validating 100% migration accuracy. Develop and maintain CI/CD pipelines for Python code deployment using Azure DevOps, checking in all converted code with comprehensive documentation. Integrate the Databricks lakehouse with downstream applications (such as Power BI and ESRI) and configure SSO integrations. Access legacy enclave environments for data profiling and side-by-side validation while collaborating daily with database managers and analysts. Mentor mid-level engineering staff on platform operations and migration methodologies, participate in Agile ceremonies, and support system acceptance demonstrations. Performs other duties as assigned. Security Clearance Requirements: Public Trust / Tier 4 Eligible: No clearance required to apply; must be a U.S. citizen willing to undergo a background check to obtain a Public Trust / Tier 4 clearance. Experience Requirements: Minimum experience required varies by degree level: PhD with 4 years; Master's degree with 8 years; Bachelor's degree with 10 years; or High School Diploma with 14 years of relevant experience. Hands-on experience with Databricks (workspace administration, notebook development, job scheduling) and proficiency in Python, PySpark, and SQL for large-scale pipeline development. Experience migrating data from legacy platforms (Hive, Hadoop, Oracle) to cloud-native platforms, utilizing Delta Lake or Apache Iceberg table formats. Practical experience with AWS services (S3, IAM, RDS, GovCloud), CI/CD pipeline tools (Azure DevOps, Jenkins, GitHub Actions), and enterprise ETL/ELT frameworks. Education Requirements: Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related technical field (or equivalent combination of education and experience). Desired Skills / Qualifications: Databricks Certified Data Engineer (Associate or Professional). Direct experience converting Informatica PowerCenter mappings to Python/PySpark code

Experience with Unity Catalog data governance, Cloudera Hive/HiveQL, Power BI/ESRI integrations, and CDC methodologies. Familiarity with FISMA High or FedRAMP High security requirements and SAML-based SSO/Okta integrations. Location: Washington, DC Status: Full time Schedule: Day shift, MondayFriday Physical Requirements: Must be able to remain in a stationary position for long durations of time. Also, must be able to continuously operate a computer and other office productivity machinery. Travel Required: No This job description is subject to change at any time. We are an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by law.