Data Pipeline Engineer: Delta Lake on AWS & Databricks

OpenKrill; United States; Remote; USD 63,000 - 92,000; Full time

Benefits offered by this job

Comprehensive medical, dental, vision

Telehealth services

3.5 weeks PTO + 11 holidays

401(k) with 4% match

Life and disability insurance

Tuition reimbursement up to $1,500

Professional development programs

Job summary

Attain Talent is seeking a Development Engineer in the United States to design and maintain scalable data pipelines for enterprise analytics. You will build ingestion, transformation, and processing workflows across AWS and Databricks, focusing on Delta Lake lakehouse architectures for healthcare data.

You will collaborate with data architects and QA analysts, implement reusable data ingestion frameworks, and optimize pipelines for performance and cost efficiency while integrating with CI/CD and

Qualifications

• Experience designing and maintaining robust ETL/ELT pipelines for analytics.
• Proficiency in Python, SQL and PySpark within Databricks environments.
• Experience with Delta Lake and data lakehouse architectures.
• Ability to collaborate with data architects and QA analysts to define data transformation logic.
• Ability to implement reusable data ingestion frameworks and CI/CD integration.

Responsibilities

• Design, build, and maintain scalable data pipelines for analytics and reporting.
• Develop data processing workflows in Python/SQL/PySpark on Databricks.
• Contribute to governance, metadata, and lineage documentation.
• Collaborate with DevOps to integrate pipelines into CI/CD workflows.
• Perform root cause analysis of data issues across environments.

Skills

ETL/ELT pipelines

Python

SQL

PySpark

Databricks

Delta Lake

Data governance

CI/CD

GitHub

Data modeling

Tools

Databricks Repos

GitHub Actions

Job description

Attain Talent is seeking a Development Engineer in the United States to design and maintain scalable data pipelines for enterprise analytics. You will build ingestion, transformation, and processing workflows across AWS and Databricks, focusing on Delta Lake lakehouse architectures for healthcare data.

You will collaborate with data architects and QA analysts, implement reusable data ingestion frameworks, and optimize pipelines for performance and cost efficiency while integrating with CI/CD and