Senior Data Engineer: Lakehouse Migrations & Databricks Lead

Agile5 Technologies, Inc.; Fairmont (WV); On-site; USD 140,000 - 200,000; Full time

Job summary

Agile5 Technologies, Inc. is seeking a Senior Data Engineer to lead complex migration projects and build scalable data pipelines in a secure AWS GovCloud Databricks environment.

You will convert legacy ETL artifacts to Python/PySpark, migrate Hive to Delta Lake, implement CDC pipelines, and govern data with Unity Catalog while mentoring engineers and collaborating across teams. A strong background in Python, PySpark, SQL, and CI/CD is required.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, IT, or a related technical field.
• Hands-on experience with Databricks (workspace administration, notebook development, job scheduling) and proficiency in Python, PySpark, and SQL.
• Experience migrating data from legacy platforms (Hive, Hadoop, Oracle) to cloud-native platforms (Delta Lake/Athena).
• Experience with AWS services (S3, IAM, GovCloud) and CI/CD tools (Azure DevOps, Jenkins, GitHub Actions).

Responsibilities

• Lead daily migration operations, data profiling, schema analysis, and deployment validation.
• Migrate Hive tables to Delta Lake format on AWS S3 using Databricks tools.
• Translate Informatica mappings to Python/PySpark preserving business logic and data quality checks.
• Design CDC pipelines to synchronize data during migration.
• Configure Databricks workspaces, clusters, notebooks, and jobs in AWS GovCloud.
• Implement Unity Catalog governance, RBAC, encryption, audit trails, and data lineage.
• Build and run automated data reconciliation scripts to ensure migration accuracy.
• Develop CI/CD pipelines for Python deployment using Azure DevOps and related tooling.
• Mentor engineers, participate in Agile ceremonies, and support demonstrations.

Skills

Python

PySpark

SQL

Data migration

Leadership

Education

Bachelor's degree in Computer Science, Data Engineering, IT, or related field

Tools

Databricks

AWS GovCloud

Azure DevOps

Jenkins

GitHub Actions

Power BI

ESRI

Job description

Agile5 Technologies, Inc. is seeking a Senior Data Engineer to lead complex migration projects and build scalable data pipelines in a secure AWS GovCloud Databricks environment.

You will convert legacy ETL artifacts to Python/PySpark, migrate Hive to Delta Lake, implement CDC pipelines, and govern data with Unity Catalog while mentoring engineers and collaborating across teams. A strong background in Python, PySpark, SQL, and CI/CD is required.