Reference Data Engineer

DataJobs; New York (NY); On-site; USD 135,000 - 170,000; Full time

Job summary

KKR’s Operations Systems team is hiring an Assistant Vice President of Software Engineering in New York, onsite. The role focuses on designing backend services and data pipelines using Python, Spark, and Terraform to enable analytics for internal stakeholders.

You will own end-to-end delivery from design to production and operations, driving reliability, performance, and observability across ETL/ELT workflows and data models. Collaboration with product and data leaders is essential.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field.
• 4–7 years of professional software engineering experience with a strong backend focus.
• Expert-level Python proficiency for production services and data pipelines.
• Hands-on experience with Apache Spark for distributed data processing at scale.
• Proficiency with Terraform for infrastructure-as-code and cloud resource management.
• Strong data engineering fundamentals, including data modeling, ETL/ELT design, and data distribution patterns.

Responsibilities

• Design, build, and own scalable backend services and data pipelines primarily in Python.
• Develop and optimize large-scale data processing workflows using Spark for batch and near-real-time use cases.
• Lead data modeling and schema design decisions to support reliable analytics and downstream consumption.
• Architect and maintain robust ETL/ELT workflows with strong data quality, data lineage, and observability.
• Design and implement data distribution patterns that enable efficient data serving across internal systems and consumers.
• Provision and manage cloud infrastructure using Terraform and infrastructure-as-code best practices.
• Own significant workstreams end-to-end, including production delivery and operations.
• Mentor and support analysts and junior engineers on the team.
• Partner with product, data, and platform leaders to shape the roadmap and translate requirements into scalable solutions.
• Lead root-cause analysis for complex production issues and drive continuous improvements in reliability and performance.

Skills

Python

Distributed processing

Data modeling

SQL

CI/CD

Cloud

Education

Bachelor’s degree in Computer Science/Engineering

Tools

Apache Spark

Terraform

Git

Airflow

Dagster

Kafka

Docker

Kubernetes

Palantir Foundry

Job description

As an Assistant Vice President of Software Engineering within KKR’s Operations Systems team, this role focuses on designing and delivering reliable backend services and data pipelines that support analytics across internal stakeholders. The position is based in New York, NY (onsite) and centers on Python, distributed processing with Apache Spark, and infrastructure-as-code using Terraform.

Role Summary

You will design and take ownership of backend systems and data engineering workflows, with emphasis on end-to-end delivery from technical design through production and operations. The work spans ETL and ELT, data modeling and schema design, data distribution patterns, and operational excellence through observability, lineage, and data quality practices.

Key Responsibilities

• Design, build, and own scalable backend services and data pipelines primarily in Python.
• Develop and optimize large-scale data processing workflows using Apache Spark for batch and near-real-time use cases.
• Lead data modeling and schema design decisions to support reliable analytics and downstream consumption.
• Architect and maintain robust ETL/ELT workflows with strong data quality, data lineage, and observability.
• Design and implement data distribution patterns that enable efficient data serving across internal systems and consumers.
• Provision and manage cloud infrastructure using Terraform and infrastructure-as-code best practices.
• Own significant workstreams end-to-end, including production delivery and operations.
• Drive engineering quality through design reviews, code reviews, and clear technical standards.
• Mentor and support analysts and junior engineers on the team.
• Partner with product, data, and platform leaders to shape the roadmap and translate requirements into scalable solutions.
• Lead root-cause analysis for complex production issues and drive continuous improvements in reliability and performance.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 4–7 years of professional software engineering experience with a strong backend focus.
• Expert-level Python proficiency for production services and data pipelines.
• Hands‑on experience with Apache Spark for distributed data processing at scale.
• Proficiency with Terraform for infrastructure‑as‑code and cloud resource management.
• Strong data engineering fundamentals, including data modeling, ETL/ELT design, and data distribution patterns.
• Solid command of relational and non‑relational data stores and SQL.
• Experience with Git, CI/CD pipelines, and modern software development practices.
• Ability to own complex workstreams and mentor other engineers.
• Strong communication skills and the ability to collaborate across technical and business teams.

Technologies

• Python
• Apache Spark
• Terraform
• ETL/ELT
• SQL
• Git
• CI/CD
• Relational data stores
• Non‑relational data stores

Preferred Qualifications

• Experience with cloud platforms (AWS, Azure, or GCP) and their data services.
• Familiarity with workflow orchestration tools (for example, Airflow, Dagster).
• Experience with streaming technologies (for example, Kafka, Spark Structured Streaming).
• Exposure to containerization and orchestration (Docker, Kubernetes).
• Prior experience in financial services or another data-intensive, regulated industry.
• Experience with Palantir Foundry or similar data integration and analytics platforms.

Compensation and Benefits

Salary range: USD 135,000 - 170,000 per year.

• Competitive compensation
• Comprehensive benefits
• Discretionary bonus, based on factors such as individual and team performance
• Opportunities for professional growth, technical leadership, and impact

How You Can Contribute

This position supports KKR’s Operations Systems through robust backend engineering and data pipeline delivery. You will focus on reliability, performance, and operational readiness, while collaborating with cross-functional partners to translate requirements into scalable solutions.