Senior Data Engineer - AI Pipelines & Financial Data

Carlyle Group; New York (NY); Hybrid; USD 150,000 - 200,000; Full time

Benefits offered by this job

Health insurance

Retirement benefits

Paid time off

Annual discretionary incentive

Job summary

Carlyle Group in New York, NY is seeking a Data Engineer to build robust data pipelines from multiple vendors to gold tables for the firm's ML team, utilizing Snowflake and Databricks platforms. The role requires strong Python and SQL skills, familiarity with financial data sources, and the ability to deploy scalable data solutions.

The position offers a hybrid work model in New York with travel up to 20%, and a base salary in the $150k–$200k range plus benefits and potential incentive programs.

Qualifications

• Bachelor's degree in CS, math, engineering or related field.
• At least 6 years of data engineering experience, preferably in financial services.
• Strong Python/SQL, data manipulation, and ML data prep fundamentals.

Responsibilities

• Build, scale, and maintain robust data pipelines and systems.
• Develop ETL/ELT processes with scalable, reliable architectures.
• Lead end-to-end software projects involving LLMs, RAG, and AI tech.
• Adopt CI/CD, IaC, containerization, and cloud-native deployments.
• Collaborate with stakeholders to turn use cases into production services.
• Implement testing and monitoring to ensure data quality and integrity.
• Mentor junior engineers and foster continuous learning.
• Travel up to 20% to collaborate with distributed teams.

Skills

Python

SQL

Databricks

PySpark

dbt

Airflow

Temporal

Prompt engineering

RAG

Vector databases

Knowledge graphs

Education

Bachelor's degree in Computer Science or related field

Tools

Snowflake

Databricks

Airflow

Temporal

dbt

Job description

Carlyle Group in New York, NY is seeking a Data Engineer to build robust data pipelines from multiple vendors to gold tables for the firm's ML team, utilizing Snowflake and Databricks platforms. The role requires strong Python and SQL skills, familiarity with financial data sources, and the ability to deploy scalable data solutions.

The position offers a hybrid work model in New York with travel up to 20%, and a base salary in the $150k–$200k range plus benefits and potential incentive programs.