Associate Data Engineer - Real-Time Data & AI Pipelines

New York Life Insurance Co; New York (NY); On-site; USD 60,000 - 86,000; Full time

Job summary

New York Life Insurance Co in New York City is seeking an Associate - Data Engineer to help build and maintain dbt data pipelines that deliver clean, reliable data for analytics and AI use cases.

You will implement real-time data ingestion with PySpark and AWS Glue, apply software engineering best practices, and collaborate with data scientists across enterprise to enable advanced analytics and data governance.

Qualifications

• 2-5+ years of hands-on experience building and maintaining dbt data models and transformation pipelines.
• Proficiency with AWS Glue, including ETL development, job orchestration, and catalog integration.
• Strong working knowledge of Databricks, including SQL, notebooks, and distributed data processing.
• Experience with Kafka and streaming data pipelines for real-time ingestion and processing.
• Familiarity with Google Cloud Platform (GCP) is a plus.
• Strong analytical thinking and problem-solving skills, with the ability to troubleshoot complex data issues.
• Ability to work collaboratively in cross-functional teams and communicate technical concepts to non-technical stakeholders.
• Experience in the insurance industry or other regulated industries (preferred)

Responsibilities

• Build and maintain dbt data pipelines to deliver clean, reliable, and well‑structured data for data science and analytics use cases.
• Implement real‑time data ingestion and processing using PySpark and AWS Glue.
• Apply software engineering best practices—including version control, testing, and integration—to all analytics and data development work.
• Review code changes and approve pull requests (PRs) in Git to ensure quality and consistency.
• Partner with stakeholders to explore, analyze, and source data from strategic enterprise data assets.
• Collaborate closely with data scientists on data quality, availability, and business value to support model development and analysis.
• Translate business and technical requirements into Jira stories, define milestones, and manage progress.
• Work with data stewards across New York Life to ensure alignment on data governance and standards.
• Build strong relationships with Technology (IT) teams to support tooling, data strategy, system integrations, and deployment processes.
• Communicate information and ideas effectively to diverse audiences across the organization.

Skills

dbt pipelines

AWS Glue

Databricks

Kafka

Jira

Tools

Git

Jira

Job description

New York Life Insurance Co in New York City is seeking an Associate - Data Engineer to help build and maintain dbt data pipelines that deliver clean, reliable data for analytics and AI use cases.

You will implement real-time data ingestion with PySpark and AWS Glue, apply software engineering best practices, and collaborate with data scientists across enterprise to enable advanced analytics and data governance.