Principal Data Engineer

Motion Recruitment Partners LLC; Waltham (MA); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Hybrid Work Environment

Medical Insurance

401(k) with company match

Paid Time Off

Job summary

Motion Recruitment Partners LLC seeks a Principal Data Engineer to lead real-time data platform design and modernization in a hybrid Waltham, MA team. You will architect scalable, cloud-native streaming systems, mentor engineers, and drive best practices across the organization.

The role emphasizes hands-on engineering, cross-functional collaboration, and guiding architecture decisions while delivering enterprise-grade data solutions for financial tech apps.

Qualifications

• 8+ years of experience in data engineering, software engineering, or backend platform development.
• Expert-level Python development for data processing, automation, and scalable backend applications.
• Advanced SQL skills including complex query development, CTEs, window functions, performance tuning, and query optimization.
• Extensive experience designing and building scalable ETL/ELT pipelines and distributed data processing systems.
• Strong experience with cloud-native data platforms on AWS, including Redshift, S3, Lambda, Glue, and related services.
• Experience with Apache Kafka and event-driven data architectures.
• Experience building and orchestrating batch and streaming data pipelines using tools such as Airflow, Spark, or Flink.
• Strong knowledge of relational databases including PostgreSQL, Oracle, Amazon Redshift, and Aurora, with expertise in schema design and data modeling.
• Experience with containerization and Infrastructure as Code using Docker, Kubernetes, Terraform, or CloudFormation
• Experience building CI/CD pipelines and implementing engineering best practices for production data platforms.
• Experience implementing monitoring, logging, data quality, and observability solutions across enterprise data environments.
• Proven ability to lead technical initiatives, mentor engineers, and collaborate across cross-functional engineering teams.

Responsibilities

• 40% Real-Time Data Platform & Streaming Engineering
• 30% Cloud Infrastructure, Microservices & Distributed Systems
• 20% Architecture, Technical Leadership & Platform Modernization
• 10% Observability, Performance Optimization & CI/CD
• 60% Hands On Engineering
• 20% Technical Leadership & Mentorship
• 20% Cross-Functional Collaboration

Skills

Python

SQL

ETL/ELT pipelines

AWS cloud-native

Kafka

Airflow/Spark/Flink

Data modeling

PostgreSQL/Oracle/Aurora

Docker/Kubernetes

Terraform/CloudFormation

CI/CD & observability

Tools

Airflow

Spark

Flink

Docker

Kubernetes

Terraform

CloudFormation

Redshift

S3/Lambda/Glue

Job description

Job Description

A global financial technology leader is hiring a Principal Data Engineer to join its hybrid engineering team in Waltham, MA. This full-time role focuses on designing and building enterprise-scale, real-time data platforms that power mission-critical financial applications.

This is a highly visible technical leadership opportunity where you'll drive the architecture and modernization of next-generation streaming data platforms while mentoring engineers and influencing engineering standards across the organization. The team is looking for someone who thrives in complex distributed environments, enjoys solving large-scale performance challenges, and wants to build resilient, cloud-native systems using cutting-edge event streaming technologies.

Required Skills & Experience

• 8+ years of experience in data engineering, software engineering, or backend platform development
• Expert-level Python development for data processing, automation, and scalable backend applications
• Advanced SQL skills including complex query development, CTEs, window functions, performance tuning, and query optimization
• Extensive experience designing and building scalable ETL/ELT pipelines and distributed data processing systems
• Strong experience with cloud-native data platforms on AWS, including Redshift, S3, Lambda, Glue, and related services
• Experience with Apache Kafka and event-driven data architectures
• Experience building and orchestrating batch and streaming data pipelines using tools such as Airflow, Spark, or Flink
• Strong knowledge of relational databases including PostgreSQL, Oracle, Amazon Redshift, and Aurora, with expertise in schema design and data modeling
• Experience with containerization and Infrastructure as Code using Docker, Kubernetes, Terraform, or CloudFormationExperience building CI/CD pipelines and implementing engineering best practices for production data platforms
• Experience implementing monitoring, logging, data quality, and observability solutions across enterprise data environments
• Proven ability to lead technical initiatives, mentor engineers, and collaborate across cross-functional engineering teams

What You Will Be Doing (Tech Breakdown)

• 40% Real-Time Data Platform & Streaming Engineering
• 30% Cloud Infrastructure, Microservices & Distributed Systems
• 20% Architecture, Technical Leadership & Platform Modernization
• 10% Observability, Performance Optimization & CI/CD

Daily Responsibilities

• 60% Hands On Engineering
• 20% Technical Leadership & Mentorship
• 20% Cross-Functional Collaboration

The Offer

You will receive the following benefits:

• Medical Insurance
• Dental Benefits
• Vision Benefits
• 401(k) with company match
• Paid Time Off (PTO)
• Paid Holidays
• Paid Parental Leave
• Professional Development Reimbursement
• Hybrid Work Environment

Applicants must be currently authorized to work in the US on a full-time basis now and in the future.