Senior Data Engineer: Databricks, Spark & Cloud Pipelines

New York Technology Partners

Irvine (CA)

On-site

USD 140,000 - 190,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

New York Technology Partners in Irvine is seeking a hands-on Senior Data Engineer to build and support enterprise-scale data solutions using Databricks and modern cloud data technologies. You will develop scalable ETL/ELT pipelines with Databricks, PySpark, Python and SQL, optimize Spark workloads, and orchestrate workflows using Airflow and Astronomer.

Partner with data scientists and business teams to deliver reliable production data environments on cloud platforms (AWS/Azure/GCP) with CI/CD

Qualifications

7+ years of Data Engineering experience.
Hands-on Databricks and Apache Spark experience.
Advanced PySpark and SQL skills.
Strong Apache Airflow and Astronomer experience.
Experience with Terraform, Pulumi, or similar infrastructure-as-code tools.
Experience building production-grade ETL/ELT pipelines.
Cloud experience with AWS, Azure, or GCP.
Experience with Git, CI/CD, monitoring, and production support.

Responsibilities

Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL
Develop and optimize batch and streaming data workflows
Create and manage Airflow DAGs and workflows using Astronomer
Optimize Spark workloads and troubleshoot production pipeline issues
Implement data quality, monitoring, and reliability standards
Support CI/CD, infrastructure automation, and production deployments
Partner with technical and business teams to deliver scalable data solutions