AVP Data Engineering: Python, Spark & Pipelines

DataJobs; New York (NY); On-site; USD 135,000 - 170,000; Full time

Job summary

KKR’s Operations Systems team is hiring an Assistant Vice President of Software Engineering in New York, onsite. The role focuses on designing backend services and data pipelines using Python, Spark, and Terraform to enable analytics for internal stakeholders.

You will own end-to-end delivery from design to production and operations, driving reliability, performance, and observability across ETL/ELT workflows and data models. Collaboration with product and data leaders is essential.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field.
• 4–7 years of professional software engineering experience with a strong backend focus.
• Expert-level Python proficiency for production services and data pipelines.
• Hands-on experience with Apache Spark for distributed data processing at scale.
• Proficiency with Terraform for infrastructure-as-code and cloud resource management.
• Strong data engineering fundamentals, including data modeling, ETL/ELT design, and data distribution patterns.

Responsibilities

• Design, build, and own scalable backend services and data pipelines primarily in Python.
• Develop and optimize large-scale data processing workflows using Spark for batch and near-real-time use cases.
• Lead data modeling and schema design decisions to support reliable analytics and downstream consumption.
• Architect and maintain robust ETL/ELT workflows with strong data quality, data lineage, and observability.
• Design and implement data distribution patterns that enable efficient data serving across internal systems and consumers.
• Provision and manage cloud infrastructure using Terraform and infrastructure-as-code best practices.
• Own significant workstreams end-to-end, including production delivery and operations.
• Mentor and support analysts and junior engineers on the team.
• Partner with product, data, and platform leaders to shape the roadmap and translate requirements into scalable solutions.
• Lead root-cause analysis for complex production issues and drive continuous improvements in reliability and performance.

Skills

Python

Distributed processing

Data modeling

SQL

CI/CD

Cloud

Education

Bachelor’s degree in Computer Science/Engineering

Tools

Apache Spark

Terraform

Git

Airflow

Dagster

Kafka

Docker

Kubernetes

Palantir Foundry

Job description

KKR’s Operations Systems team is hiring an Assistant Vice President of Software Engineering in New York, onsite. The role focuses on designing backend services and data pipelines using Python, Spark, and Terraform to enable analytics for internal stakeholders.

You will own end-to-end delivery from design to production and operations, driving reliability, performance, and observability across ETL/ELT workflows and data models. Collaboration with product and data leaders is essential.