On-Site Senior Data Engineer: Databricks & Delta Lake Lead

Agile5 Technologies, Inc.; Northern (KY); Hybrid; USD 90,000 - 165,000; Full time

Job summary

Agile5 Technologies, Inc. is seeking a Senior Data Engineer on site in Washington, DC. This role leads data migration and pipeline modernization, with focus on Databricks, Python/PySpark, and governance within a secure AWS GovCloud environment.

You will oversee ingestion, schema analysis, and production validation while mentoring engineers and collaborating with DBAs and analysts to ensure 100% data accuracy.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, IT, or related field (or equivalent).
• Hands-on experience with Databricks (workspace admin, notebook development, job scheduling).
• Experience migrating data from Hive/Hadoop/Oracle to cloud-native platforms (Delta Lake/ Iceberg).
• Proficiency in Python, PySpark, and SQL for large-scale pipelines.
• Public Trust / Tier 4 eligible: US citizen; background check required.

Responsibilities

• Lead data migration and pipeline engineering for lakehouse modernization.
• Migrate legacy Hive/Hadoop/ETL artifacts to Python/PySpark.
• Design and implement CDC pipelines for data synchronization.
• Manage Databricks environment, CI/CD pipelines, and governance tooling.
• Collaborate with DBAs and analysts, mentor mid-level engineers.

Skills

Python

PySpark

SQL

Data governance

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Azure DevOps

Jenkins

GitHub Actions

AWS GovCloud

Job description

Agile5 Technologies, Inc. is seeking a Senior Data Engineer on site in Washington, DC. This role leads data migration and pipeline modernization, with focus on Databricks, Python/PySpark, and governance within a secure AWS GovCloud environment.

You will oversee ingestion, schema analysis, and production validation while mentoring engineers and collaborating with DBAs and analysts to ensure 100% data accuracy.