Senior Data Engineer - Remote | Databricks & Lakehouse

Quantiphi; United States; Remote; USD 140,000 - 200,000; Full time

Job summary

Quantiphi is seeking a Senior Data Engineer to design and optimize real-time and batch data pipelines using Python, PySpark, and Databricks. Strong SQL and cloud experience across GCP/AWS/Azure are required for this remote role aligned to US business hours.

You will collaborate with stakeholders to translate business needs into scalable data architectures, implement Delta Lake and lakehouse patterns, and contribute to CI/CD pipelines with Terraform and GitHub Actions.

Qualifications

• Strong Python, SQL and PySpark skills with real-time and batch pipelines.
• Expertise in Databricks, Delta Lake, and data lakehouse concepts.
• Cloud experience across GCP/AWS/Azure and CI/CD with Terraform/GitHub Actions.

Responsibilities

• Design, build, and optimize scalable ETL/ELT data pipelines for TB-scale datasets.
• Manage Databricks workflows, clusters, Unity Catalog, and Delta Lake.
• Lead cloud migrations to cloud storage and data warehouses; implement security controls.
• Develop data-driven APIs and scripts with Python and Flask; support DevOps automation.

Skills

Python

SQL

PySpark

Databricks

Terraform

GitHub Actions

Docker

Flask

Cloud platforms

Job description

Quantiphi is seeking a Senior Data Engineer to design and optimize real-time and batch data pipelines using Python, PySpark, and Databricks. Strong SQL and cloud experience across GCP/AWS/Azure are required for this remote role aligned to US business hours.

You will collaborate with stakeholders to translate business needs into scalable data architectures, implement Delta Lake and lakehouse patterns, and contribute to CI/CD pipelines with Terraform and GitHub Actions.