Senior Data Engineer

New York Technology Partners; Irvine (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

New York Technology Partners in Irvine is seeking a hands-on Senior Data Engineer to build and support enterprise-scale data solutions using Databricks and modern cloud data technologies. You will develop scalable ETL/ELT pipelines with Databricks, PySpark, Python and SQL, optimize Spark workloads, and orchestrate workflows using Airflow and Astronomer.

Partner with data scientists and business teams to deliver reliable production data environments on cloud platforms (AWS/Azure/GCP) with CI/CD

Qualifications

• 7+ years of Data Engineering experience.
• Hands-on Databricks and Apache Spark experience.
• Advanced PySpark and SQL skills.
• Strong Apache Airflow and Astronomer experience.
• Experience with Terraform, Pulumi, or similar infrastructure-as-code tools.
• Experience building production-grade ETL/ELT pipelines.
• Cloud experience with AWS, Azure, or GCP.
• Experience with Git, CI/CD, monitoring, and production support.

Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL
• Develop and optimize batch and streaming data workflows
• Create and manage Airflow DAGs and workflows using Astronomer
• Optimize Spark workloads and troubleshoot production pipeline issues
• Implement data quality, monitoring, and reliability standards
• Support CI/CD, infrastructure automation, and production deployments
• Partner with technical and business teams to deliver scalable data solutions

Skills

Databricks

PySpark

Python

SQL

Airflow

Astronomer

Terraform

Pulumi

CI/CD

Git

Monitoring

Production support

Tools

Databricks

Apache Spark

Airflow

Astronomer

Terraform

Pulumi

Job description

We're looking for a hands-on Senior Data Engineer to build and support enterprise-scale data solutions using Databricks and modern cloud data technologies. This role will focus on developing scalable data pipelines, optimizing Spark workloads, orchestrating workflows, and supporting reliable production data environments.

Key Responsibilities

• Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL
• Develop and optimize batch and streaming data workflows
• Create and manage Airflow DAGs and workflows using Astronomer
• Optimize Spark workloads and troubleshoot production pipeline issues
• Implement data quality, monitoring, and reliability standards
• Support CI/CD, infrastructure automation, and production deployments
• Partner with technical and business teams to deliver scalable data solutions

Required Experience

• 7+ years of Data Engineering experience
• Strong hands-on Databricks and Apache Spark experience
• Advanced PySpark and SQL skills
• Strong Apache Airflow and Astronomer experience
• Experience with Terraform, Pulumi, or similar infrastructure-as-code tools
• Experience building production-grade ETL/ELT pipelines
• Cloud experience with AWS, Azure, or GCP
• Experience with Git, CI/CD, monitoring, and production support

Nice to Have

• Delta Lake and Unity Catalog
• dbt
• Kafka or Structured Streaming
• Financial services or other regulated industry experience