Principal Data Engineer

Octus; New York (NY); On-site; USD 180,000 - 230,000; Full time

Benefits offered by this job

Competitive health benefits

Matched 401k and pension

PTO

Generous parental leave

Gym subsidies

Educational reimbursements

Recognition programs

Job summary

Octus is seeking a Principal Data Engineer to design, develop, and lead scalable data ingestion and transformation pipelines. This is a hands-on IC role with some mentoring and architecture guidance, contributing to robust data infrastructure powering platforms and automation across the firm.

You will partner with data analysts, scientists and engineers to ensure data quality, lineage, and reliability while applying best practices in data modeling, ETL/ELT, and cloud-native design on AWS.

Qualifications

• 8+ years of experience in data engineering or backend development.
• Expert proficiency in Python for data pipeline and automation development.
• Advanced SQL skills and experience optimizing complex queries and data models.
• Experience designing cloud-native data pipelines on AWS (MWAA/Airflow, Lambda, ECS, SQS, Glue, S3, Redshift).
• Experience with data warehousing or lakehouse technologies (Redshift, Snowflake, Databricks).
• Experience implementing Terraform or similar IaC frameworks.
• Strong understanding of data ingestion, transformation, and orchestration tools and patterns, including AI/ML pipelines.
• Familiarity with CI/CD pipelines, automated testing, and modern DevOps practices.
• 8+ years of experience in data engineering or backend development, with a focus on scalable data solutions.
• Extensive experience in a technical leadership capacity, including mentoring and leading complex data infrastructure projects end-to-end.
• Familiarity with containerization (Docker) and workflow orchestration best practices.
• Excellent communication, collaboration, and problem-solving skills.

Responsibilities

• Lead the overarching technical strategy for the data platform.
• Lead the design and development of data ingestion and transformation pipelines across diverse data sources.
• Serve as a foundational technical leader and mentor for senior engineers within the data platform team.
• Architect and manage data pipelines and orchestration workflows using AWS services (MWAA, Lambda, ECS, SQS).
• Implement and maintain infrastructure as code (IaC) using Terraform.
• Partner with data analysts, scientists, and backend engineers to ensure data consistency and reliability.
• Apply best practices in data modeling, schema design, and ETL/ELT for high-volume data.
• Ensure data quality and lineage through automated testing, monitoring, and alerting.
• Promote continuous improvement through code reviews and observability practices.
• Collaborate with technology leadership to align data platform with business strategy.
• Stay up to date with trends in data engineering, cloud architecture, AI/ML integration.

Skills

Python

SQL

AWS

Cloud data pipelines

Terraform

Docker

DevOps

Communication

Tools

MWAA/Airflow

Lambda

ECS

SQS

Redshift

S3

Glue

Databricks

Job description

Octus is a leading global provider of credit intelligence, data, and analytics. Since 2013, tens of thousands of professionals across hedge fund, investment banking, management consulting, and law firm verticals have come to rely on Octus to make better, faster, and more confident decisions in pace with the fast-moving credit markets. For more information, visit: https://octus.com/

Working at Octus

Octus hires growth-minded innovators and trailblazers across the globe to drive our business and culture. Our core values – Action Oriented, Customer First Mindset, Effective Team Players, and Driven to Excel – define an organizational ethos that’s as high-performing as it is human.

Among other perks, Octus employees enjoy:

• competitive health benefits
• matched 401k and pension plans
• PTO
• generous parental leave
• gym subsidies
• educational reimbursements for career development
• recognition programs

Role

Octus is seeking a Principal Data Engineer to design, develop, and lead scalable data ingestion and transformation pipelines. This is a hands-on Individual Contributor (IC) role that encompasses some management duties, including mentoring, architecture guidance, and providing technical leadership. You’ll play a key role in designing and maintaining robust data infrastructure that powers data platforms, products and automation initiatives across the firm. The ideal candidate is an expert Python and SQL developer with deep experience building modern data workflows using AWS services and infrastructure-as-code.

Responsibilities

• Lead the overarching technical strategy for the data platform, ensuring alignment between data infrastructure and long-term business goals.
• Lead the design and development of data ingestion and transformation pipelines, ensuring scalability, efficiency, and reliability across diverse data sources (APIs, web data, internal feeds, etc.).
• Serve as a foundational technical leader and mentor for senior engineers within the data platform team, guiding architecture, design, and implementation decisions.
• Architect and manage data pipelines and orchestration workflows using AWS services such as MWAA (Airflow), Lambda, ECS, and SQS.
• Implement and maintain infrastructure as code (IaC) using Terraform, ensuring reproducibility and compliance with cloud standards.
• Partner with data analysts, scientists, and backend engineers to ensure data consistency, discoverability, and reliability.
• Apply best practices in data modeling, schema design, and ETL/ELT processes for high-volume structured and semi-structured data.
• Ensure data quality and lineage through automated testing, monitoring, and alerting.
• Promote continuous improvement through code reviews, observability practices, and team-wide knowledge sharing.
• Collaborate closely with technology leadership to align data platform development with business strategy and product goals.
• Stay up to date with industry trends in data engineering, cloud architecture, AI/ML integration, and automation.

Requirements

• Strong foundation in software engineering principles, including SOLID design, modularity, and scalability.
• Expert proficiency in Python for data pipeline and automation development.
• Advanced SQL skills and experience optimizing complex queries and data models.
• Proven experience designing and maintaining cloud-native data pipelines on AWS (e.g., MWAA/Airflow, Lambda, ECS, SQS, Glue, S3, Redshift, etc.).
• Experience with data warehousing or lakehouse technologies (Redshift, Snowflake, Databricks, etc.).
• Experience implementing and managing Terraform or similar IaC frameworks.
• Strong understanding of data ingestion, transformation, and orchestration tools and patterns, including those used in AI/ML pipelines.
• Familiarity with CI/CD pipelines, automated testing, and modern DevOps practices.
• 8+ years of experience in data engineering or backend development, with a focus on scalable data solutions.
• Extensive experience in a technical leadership capacity, including mentoring and leading complex data infrastructure projects end-to-end.
• Familiarity with containerization (Docker) and workflow orchestration best practices.
• Excellent communication, collaboration, and problem-solving skills.

Nice to Have

• Exposure to streaming data technologies (Kafka, Kinesis, Flink).
• Experience integrating data quality and observability tools (Great Expectations, Monte Carlo, etc.).
• Familiarity with Scrapy, BeautifulSoup, or other data extraction frameworks for ingestion pipelines.

Equal Employment Opportunity

Octus is committed to providing equal employment opportunities to all employees and applicants for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, age, disability, genetic information, marital status, pregnancy, veteran status, or any other legally protected status. We strive to create an inclusive and diverse work environment where all individuals are valued, respected, and treated fairly. We believe that diversity enriches our workplace and enhances our ability to innovate and succeed.