Data Platform Engineer

Shift4; Toronto; On-site; CAD 120,000 - 160,000; Full time

Benefits offered by this job

Wellness programs

Career growth opportunities

Happy hours and events

Paid time off

Flexible work schedules

Paid leave programs

Top Gun Program

Job summary

Shift4 is seeking a Data Platform Engineer to build, maintain, and optimize data ingestion pipelines from production systems into Snowflake. You will focus on pipeline performance, reliability, scalability, cost efficiency, and observability.

Experience with ECS, SQS, Python, and Terraform is highly valued. The role centers on operating pipelines into Snowflake, defining SLOs, building alerting and dashboards, and ensuring data is delivered accurately and on time as the platform grows.

Qualifications

• 5+ years in platform, infrastructure, or DevOps for data-intensive production systems.
• Experience operating pipelines into Snowflake from the ingestion side (Snowpipe/COPY).
• Proficient Python for reading, debugging, and instrumenting worker code.
• Strong track record tuning message-driven workloads (concurrency, retries, idempotency).
• Infrastructure as code experience with Terraform and Ansible; Docker containers.
• Deep hands-on AWS experience: ECS/Fargate, SQS, S3, IAM, CloudWatch, and Aurora/RDS.
• Experience with observability tools (Prometheus, Grafana, CloudWatch, Datadog).
• PCI-compliant or regulated environment experience; payments/fintech domain knowledge.

Responsibilities

• Operate, tune, and scale ECS/SQS-based ingestion pipelines from Aurora MySQL into Snowflake.
• Define and monitor SLOs for data freshness and lag; lead incident response and RCA.
• Build IaC, deployment automation, and CI/CD for pipeline services.
• Onboard new sources and consumers to the platform with cross-team collaboration.
• Apply data-quality and security controls appropriate for PCI-compliant environments.

Skills

AWS

ECS/Fargate

SQS

Python

Terraform

Ansible

CI/CD

Docker

Prometheus

Grafana

Snowflake ingestion

Tools

Terraform

Ansible

Snowflake

Debezium

AWS DMS

Kafka/Confluent

Job description

• Shift4 is seeking a Data Platform Engineer to join our Data Platform team and build, maintain, and optimize the pipelines that move data from our production systems into our analytics warehouse.
• Today, our infrastructure uses Amazon ECS services and SQS queues to replicate data from Aurora MySQL into Snowflake, with plans to expand to additional data sources and consumers as our platform evolves
• This is a hands‑on infrastructure engineering role focused on pipeline performance, reliability, scalability, resource optimization, cost efficiency, and observability.
• You will help ensure data is delivered accurately, reliably, and on time as our data platform grows.
• Data modeling, warehouse cost management, and business transformation logic are managed by partner teams, allowing you to focus on the infrastructure and systems that power our data ecosystem—all while embracing the Shift4 way
• Operate, tune, and scale the ECS/SQS‑based ingestion pipelines from Aurora MySQL into Snowflake: task sizing, autoscaling, consumer concurrency, batching, retries, and dead‑letter handling
• Own pipeline reliability: define SLOs for freshness and lag, build alerting and dashboards, lead incident response and root‑cause analysis for pipeline failures
• Build and maintain infrastructure as code, deployment automation, and CI/CD for pipeline services
• Continuously benchmark and tune throughput versus cost across ECS, SQS, and Snowflake ingestion (Snowpipe/COPY, stages)
• Onboard new sources and consumers to the platform in partnership with application, data, and analytics teams
• Apply data‑quality and security controls appropriate for PCI‑compliant or payment‑handling environments: encryption, secrets handling, least‑privilege access, audit trails
• Proactively explore and apply AI tools to improve efficiency across the pipeline lifecycle (development, testing, runbooks, and documentation)

Benefits

• Wellness programs, including mental health resources
• Opportunities for career growth, skill-building, and advancement
• Happy hours, office celebrations, and well‑being initiatives
• Paid time off for relaxation and quality time with loved ones
• Flexible work schedules empower you to manage your commitments effectively
• Various paid leave programs, including sick leave, parental leave, and more to support life’s important moments
• Employee‑nominated “Top Gun Program” to recognize exceptional individuals

Observability and alerting for pipeline health and lag using Prometheus/Grafana, CloudWatch, Datadog, or the Elastic StackPassion for automation, performance, and clean system designDeep hands‑on AWS experience: ECS/Fargate (task sizing, autoscaling, deployments), SQS (throughput, DLQs, visibility timeouts, batching), S3, IAM, CloudWatch, and Aurora/RDS from the operator side5+ years in platform, infrastructure, or DevOps roles supporting data‑intensive systems in productionExperience operating pipelines into Snowflake from the ingestion side (Snowpipe/COPY, stages, load monitoring)Enough Python to read, debug, and instrument worker codeProven track record tuning message‑driven workloads: consumer concurrency, backpressure, retries, idempotency, throughput versus costInfrastructure as code with Terraform/Ansible, containerization with Docker, and CI/CD pipelinesCDC tooling (Debezium, AWS DMS) and MySQL Binlog replicationElastic Stack or time‑series database (Prometheus, InfluxDB) administrationMessage brokers beyond SQS: Apache Kafka/Confluent, RabbitMQExperience in a PCI or otherwise regulated environmentPayments or fintech domain knowledge