Data Platform Engineer: Scale Pipelines & Reliability

Shift4; Toronto; On-site; CAD 120,000 - 160,000; Full time

Benefits offered by this job

Wellness programs

Career growth opportunities

Happy hours and events

Paid time off

Flexible work schedules

Paid leave programs

Top Gun Program

Job summary

Shift4 is seeking a Data Platform Engineer to build, maintain, and optimize data ingestion pipelines from production systems into Snowflake. You will focus on pipeline performance, reliability, scalability, cost efficiency, and observability.

Experience with ECS, SQS, Python, and Terraform is highly valued. The role centers on operating pipelines into Snowflake, defining SLOs, building alerting and dashboards, and ensuring data is delivered accurately and on time as the platform grows.

Qualifications

• 5+ years in platform, infrastructure, or DevOps for data-intensive production systems.
• Experience operating pipelines into Snowflake from the ingestion side (Snowpipe/COPY).
• Proficient Python for reading, debugging, and instrumenting worker code.
• Strong track record tuning message-driven workloads (concurrency, retries, idempotency).
• Infrastructure as code experience with Terraform and Ansible; Docker containers.
• Deep hands-on AWS experience: ECS/Fargate, SQS, S3, IAM, CloudWatch, and Aurora/RDS.
• Experience with observability tools (Prometheus, Grafana, CloudWatch, Datadog).
• PCI-compliant or regulated environment experience; payments/fintech domain knowledge.

Responsibilities

• Operate, tune, and scale ECS/SQS-based ingestion pipelines from Aurora MySQL into Snowflake.
• Define and monitor SLOs for data freshness and lag; lead incident response and RCA.
• Build IaC, deployment automation, and CI/CD for pipeline services.
• Onboard new sources and consumers to the platform with cross-team collaboration.
• Apply data-quality and security controls appropriate for PCI-compliant environments.

Skills

AWS

ECS/Fargate

SQS

Python

Terraform

Ansible

CI/CD

Docker

Prometheus

Grafana

Snowflake ingestion

Tools

Terraform

Ansible

Snowflake

Debezium

AWS DMS

Kafka/Confluent

Job description

Shift4 is seeking a Data Platform Engineer to build, maintain, and optimize data ingestion pipelines from production systems into Snowflake. You will focus on pipeline performance, reliability, scalability, cost efficiency, and observability.

Experience with ECS, SQS, Python, and Terraform is highly valued. The role centers on operating pipelines into Snowflake, defining SLOs, building alerting and dashboards, and ensuring data is delivered accurately and on time as the platform grows.