Senior Data Engineer & Platform Architect (AWS)

Octus; New York (NY); On-site; USD 180,000 - 230,000; Full time

Benefits offered by this job

Competitive health benefits

Matched 401k and pension

PTO

Generous parental leave

Gym subsidies

Educational reimbursements

Recognition programs

Job summary

Octus is seeking a Principal Data Engineer to design, develop, and lead scalable data ingestion and transformation pipelines. This is a hands-on IC role with some mentoring and architecture guidance, contributing to robust data infrastructure powering platforms and automation across the firm.

You will partner with data analysts, scientists and engineers to ensure data quality, lineage, and reliability while applying best practices in data modeling, ETL/ELT, and cloud-native design on AWS.

Qualifications

• 8+ years of experience in data engineering or backend development.
• Expert proficiency in Python for data pipeline and automation development.
• Advanced SQL skills and experience optimizing complex queries and data models.
• Experience designing cloud-native data pipelines on AWS (MWAA/Airflow, Lambda, ECS, SQS, Glue, S3, Redshift).
• Experience with data warehousing or lakehouse technologies (Redshift, Snowflake, Databricks).
• Experience implementing Terraform or similar IaC frameworks.
• Strong understanding of data ingestion, transformation, and orchestration tools and patterns, including AI/ML pipelines.
• Familiarity with CI/CD pipelines, automated testing, and modern DevOps practices.
• 8+ years of experience in data engineering or backend development, with a focus on scalable data solutions.
• Extensive experience in a technical leadership capacity, including mentoring and leading complex data infrastructure projects end-to-end.
• Familiarity with containerization (Docker) and workflow orchestration best practices.
• Excellent communication, collaboration, and problem-solving skills.

Responsibilities

• Lead the overarching technical strategy for the data platform.
• Lead the design and development of data ingestion and transformation pipelines across diverse data sources.
• Serve as a foundational technical leader and mentor for senior engineers within the data platform team.
• Architect and manage data pipelines and orchestration workflows using AWS services (MWAA, Lambda, ECS, SQS).
• Implement and maintain infrastructure as code (IaC) using Terraform.
• Partner with data analysts, scientists, and backend engineers to ensure data consistency and reliability.
• Apply best practices in data modeling, schema design, and ETL/ELT for high-volume data.
• Ensure data quality and lineage through automated testing, monitoring, and alerting.
• Promote continuous improvement through code reviews and observability practices.
• Collaborate with technology leadership to align data platform with business strategy.
• Stay up to date with trends in data engineering, cloud architecture, AI/ML integration.

Skills

Python

SQL

AWS

Cloud data pipelines

Terraform

Docker

DevOps

Communication

Tools

MWAA/Airflow

Lambda

ECS

SQS

Redshift

S3

Glue

Databricks

Job description

Octus is seeking a Principal Data Engineer to design, develop, and lead scalable data ingestion and transformation pipelines. This is a hands-on IC role with some mentoring and architecture guidance, contributing to robust data infrastructure powering platforms and automation across the firm.

You will partner with data analysts, scientists and engineers to ensure data quality, lineage, and reliability while applying best practices in data modeling, ETL/ELT, and cloud-native design on AWS.