Senior Data Engineer

Selby Jennings; New York (NY); On-site; USD 150,000 - 185,000; Full time

Job summary

Selby Jennings is seeking an experienced Senior Data Engineer to build and optimize scalable data pipelines and data products in a dynamic environment in New York. You will develop reliable datasets for analytics, reporting, and AI applications, establishing standards and best practices across the data platform.

The role requires strong Python and SQL skills, hands-on work with Databricks and Snowflake, and deep knowledge of data modeling, ETL/ELT, and data warehousing concepts.

Qualifications

• 6+ years of data engineering experience in production environments.
• Advanced Python and SQL skills.
• Strong experience with Databricks, Snowflake, or similar cloud data platforms.
• Hands-on experience with Spark/PySpark and large-scale data processing.
• Deep understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience building data quality, validation, and monitoring solutions.
• Comfortable working with complex, imperfect datasets and solving data integrity challenges.
• Strong communication skills and ability to operate independently.
• Bachelor's degree in a technical or quantitative field.

Responsibilities

• Build and optimize scalable data pipelines and data products.
• Develop reliable datasets for analytics, reporting, and AI applications.
• Establish data engineering standards, frameworks, and best practices.
• Improve data quality, monitoring, testing, and platform observability.
• Integrate and reconcile complex datasets from multiple sources.
• Partner with business stakeholders and technical teams to deliver impactful solutions.
• Drive automation and operational excellence across the data platform.
• Mentor engineers and contribute to technical leadership within the team.

Skills

Data engineering

Python

SQL

Databricks

Snowflake

Spark/PySpark

Data modeling

ETL/ELT

Data quality & monitoring

Communication

Education

Bachelor's degree in a technical or quantitative field

Tools

CI/CD

Terraform

Job description

Responsibilities

• Build and optimize scalable data pipelines and data products.
• Develop reliable datasets for analytics, reporting, and AI applications.
• Establish data engineering standards, frameworks, and best practices.
• Improve data quality, monitoring, testing, and platform observability.
• Integrate and reconcile complex datasets from multiple sources.
• Partner with business stakeholders and technical teams to deliver impactful solutions.
• Drive automation and operational excellence across the data platform.
• Mentor engineers and contribute to technical leadership within the team.

Qualifications

• 6+ years of data engineering experience in production environments.
• Advanced Python and SQL skills.
• Strong experience with Databricks, Snowflake, or similar cloud data platforms.
• Hands-on experience with Spark/PySpark and large-scale data processing.Deep understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience building data quality, validation, and monitoring solutions.
• Comfortable working with complex, imperfect datasets and solving data integrity challenges.
• Strong communication skills and ability to operate independently.
• Bachelor's degree in a technical or quantitative field.

Preferred

• Azure cloud experience.
• Background working with financial, healthcare, life sciences, or alternative data.
• Experience with CI/CD, Terraform, and modern data engineering tooling.
• Exposure to AI/ML platforms and data products.
• Previous leadership or mentoring experience.