Principal Data Engineer: Real-Time Cloud Platform Architect

Motion Recruitment Partners LLC; Waltham (MA); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Hybrid Work Environment

Medical Insurance

401(k) with company match

Paid Time Off

Job summary

Motion Recruitment Partners LLC seeks a Principal Data Engineer to lead real-time data platform design and modernization in a hybrid Waltham, MA team. You will architect scalable, cloud-native streaming systems, mentor engineers, and drive best practices across the organization.

The role emphasizes hands-on engineering, cross-functional collaboration, and guiding architecture decisions while delivering enterprise-grade data solutions for financial tech apps.

Qualifications

• 8+ years of experience in data engineering, software engineering, or backend platform development.
• Expert-level Python development for data processing, automation, and scalable backend applications.
• Advanced SQL skills including complex query development, CTEs, window functions, performance tuning, and query optimization.
• Extensive experience designing and building scalable ETL/ELT pipelines and distributed data processing systems.
• Strong experience with cloud-native data platforms on AWS, including Redshift, S3, Lambda, Glue, and related services.
• Experience with Apache Kafka and event-driven data architectures.
• Experience building and orchestrating batch and streaming data pipelines using tools such as Airflow, Spark, or Flink.
• Strong knowledge of relational databases including PostgreSQL, Oracle, Amazon Redshift, and Aurora, with expertise in schema design and data modeling.
• Experience with containerization and Infrastructure as Code using Docker, Kubernetes, Terraform, or CloudFormation
• Experience building CI/CD pipelines and implementing engineering best practices for production data platforms.
• Experience implementing monitoring, logging, data quality, and observability solutions across enterprise data environments.
• Proven ability to lead technical initiatives, mentor engineers, and collaborate across cross-functional engineering teams.

Responsibilities

• 40% Real-Time Data Platform & Streaming Engineering
• 30% Cloud Infrastructure, Microservices & Distributed Systems
• 20% Architecture, Technical Leadership & Platform Modernization
• 10% Observability, Performance Optimization & CI/CD
• 60% Hands On Engineering
• 20% Technical Leadership & Mentorship
• 20% Cross-Functional Collaboration

Skills

Python

SQL

ETL/ELT pipelines

AWS cloud-native

Kafka

Airflow/Spark/Flink

Data modeling

PostgreSQL/Oracle/Aurora

Docker/Kubernetes

Terraform/CloudFormation

CI/CD & observability

Tools

Airflow

Spark

Flink

Docker

Kubernetes

Terraform

CloudFormation

Redshift

S3/Lambda/Glue

Job description

Motion Recruitment Partners LLC seeks a Principal Data Engineer to lead real-time data platform design and modernization in a hybrid Waltham, MA team. You will architect scalable, cloud-native streaming systems, mentor engineers, and drive best practices across the organization.

The role emphasizes hands-on engineering, cross-functional collaboration, and guiding architecture decisions while delivering enterprise-grade data solutions for financial tech apps.