Data Engineer — Real-Time Pipelines & AI (Hybrid)

New York Life; New York (NY); Hybrid; USD 60,000 - 86,000; Full time

Benefits offered by this job

Hybrid work arrangement

Comprehensive benefits

Job summary

New York Life is seeking a data engineer to join our Technology data team, focusing on building robust dbt data pipelines and real-time processing with PySpark and AWS Glue. You'll collaborate with data scientists and stakeholders to source data, ensure quality, and translate requirements into actionable Jira stories, while upholding governance and standards across the enterprise.

Hybrid work arrangement (3 days per week) and competitive compensation accompany our focus on scalable data

Qualifications

• 2–5+ years of hands-on experience building and maintaining dbt data models and pipelines.
• Proficiency with AWS Glue ETL development, orchestration, and catalog integration.
• Strong knowledge of Databricks (SQL/notebooks) and distributed data processing.
• Experience with Kafka for real-time ingestion and processing.
• Familiarity with Google Cloud Platform (GCP) is a plus.
• Strong analytical thinking, problem-solving, and cross-functional collaboration.
• Experience in the insurance industry or regulated domains preferred.

Responsibilities

• Build and maintain dbt data pipelines delivering clean, reliable data for analytics.
• Implement real-time data ingestion using PySpark and AWS Glue.
• Apply software engineering practices across analytics and data development work.
• Review code changes and approve PRs in Git to ensure quality.
• Partner with stakeholders to source data from strategic enterprise assets.
• Collaborate with data scientists on data quality, availability, and value.
• Translate requirements into Jira stories, set milestones, manage progress.
• Work with data stewards to align on governance and standards.
• Build relationships with Technology teams to support tooling and integrations.
• Communicate effectively with diverse audiences across the organization.

Skills

dbt pipelines

AWS Glue

Databricks

Kafka

GCP

Data governance

Tools

Git

Jira

PySpark

Job description

New York Life is seeking a data engineer to join our Technology data team, focusing on building robust dbt data pipelines and real-time processing with PySpark and AWS Glue. You'll collaborate with data scientists and stakeholders to source data, ensure quality, and translate requirements into actionable Jira stories, while upholding governance and standards across the enterprise.

Hybrid work arrangement (3 days per week) and competitive compensation accompany our focus on scalable data