Data Engineer

Carlyle Group; New York, Northern (NY, KY); Hybrid; USD 150,000 - 200,000; Full time

Benefits offered by this job

Health insurance

Retirement benefits

Paid time off

Annual discretionary incentive

Job summary

Carlyle Group in New York, NY is seeking a Data Engineer to build robust data pipelines from multiple vendors to gold tables for the firm's ML team, utilizing Snowflake and Databricks platforms. The role requires strong Python and SQL skills, familiarity with financial data sources, and the ability to deploy scalable data solutions.

The position offers a hybrid work model in New York with travel up to 20%, and a base salary in the $150k–$200k range plus benefits and potential incentive programs.

Qualifications

• Bachelor's degree in CS, math, engineering or related field.
• At least 6 years of data engineering experience, preferably in financial services.
• Strong Python/SQL, data manipulation, and ML data prep fundamentals.

Responsibilities

• Build, scale, and maintain robust data pipelines and systems.
• Develop ETL/ELT processes with scalable, reliable architectures.
• Lead end-to-end software projects involving LLMs, RAG, and AI tech.
• Adopt CI/CD, IaC, containerization, and cloud-native deployments.
• Collaborate with stakeholders to turn use cases into production services.
• Implement testing and monitoring to ensure data quality and integrity.
• Mentor junior engineers and foster continuous learning.
• Travel up to 20% to collaborate with distributed teams.

Skills

Python

SQL

Databricks

PySpark

dbt

Airflow

Temporal

Prompt engineering

RAG

Vector databases

Knowledge graphs

Education

Bachelor's degree in Computer Science or related field

Tools

Snowflake

Databricks

Airflow

Temporal

dbt

Job description

## Data EngineerApply: Hybrid: New York, NY: Full time: Posted 4 Days Ago: R-00280**Position Summary**We are seeking a highly skilled Data Engineer to join our dynamic team. The ideal candidate will be responsible for creating robust data pipelines from various data vendors to gold tables, primarily for our Machine Learning (ML) team, utilizing Snowflake and Databricks platforms. The role demands expertise in Python, deep familiarity with financial data sources, and the ability to deploy complex data pipelines efficiently. This position requires a proactive approach to analyzing, aggregating, and enriching financial data from both private and public companies.**Primary Responsibilities*** Build, scale, and maintain robust data solutions to support the firm's objectives.* Implement and optimize high-performance data pipelines -- extraction, loading, transformation, and orchestration – that are designed for scalability, reliability, maintainability, and speed.* Lead software development projects end to end involving large language models (LLMs), retrieval-augmented generation (RAG) frameworks, and other AI technologies.* Champion modern software engineering practices as CI/CD, infrastructure-as-code, containerization, and cloud-native deployments* Collaborate closely with business stakeholders to transform use cases into production-ready services and solutions, owning the system from concept to production.* Implement rigorous testing and monitoring practices to maintain superior data quality and integrity.* Mentor and develop junior team members, fostering a culture of excellence and continuous learning within the team.* Be willing to travel up to 20% of the time to collaborate with distributed team members across locations.**Requirements****Education & Certificates*** A bachelor's degree, required* Concentration in Computer Science, Math, Physics, STEM or other engineering related field, preferred**Professional Experience*** At least 6 years of experience in data engineering or a related discipline, with a proven track record of success, required* Experience in the financial services or private equity industry, preferred**Competencies & Attributes*** Expertise in Python and SQL, with a strong foundation in data manipulation and analysis.* Proficient with Databricks/PySpark and dbt for data warehousing and data transformation tasks.* Experience with workflow orchestration tools e.g. Airflow, Temporal* Experience working with large language models (LLMs) especially prompt engineering, retrieval-augmented generation (RAG)s, and/or vector databases.* Knowledge of fundamental principles of machine learning, feature engineering, and knowledge graphs are pluses.* Demonstrated experience in designing and implementing complex data systems from the ground up.* Proficient in handling large-scale data projects, including data cleaning, ETL, and information retrieval.* Previous experience in a product development or financial services environment is highly desirable.* Excellent communication skills required, both verbal and written.**Benefits/Compensation** The compensation range for this role is specific to New York, NY and takes into account a wide range of factors including but not limited to the skill sets required/preferred; prior experience and training; licenses and/or certifications. The anticipated base salary range for this role is $150,000 to $200,000. In addition to the base salary, the hired professional will enjoy a comprehensive benefits package spanning retirement benefits, health insurance, life insurance and disability, paid time off, paid holidays, family planning benefits and various wellness programs. Additionally, the hired professional may also be eligible to participate in an annual discretionary incentive program, the award of which will be dependent on various factors, including, without limitation, individual and organizational performance. Due to the high volume of candidates, please be advised that only candidates selected to interview will be contacted by Carlyle.