Data Engineer

Shift4; Center Valley (Lehigh County); Vor Ort; USD 90.000 - 120.000; Vollzeit

Zusammenfassung

Shift4 is seeking a hands-on Data Engineer to build reliable SQL and Python workloads in Snowflake for payments datasets and reporting.

The role focuses on designing Python jobs, writing performant SQL, and packaging workloads in Docker for scheduled execution. Collaboration with finance and product teams is essential to translate questions into correct queries, with strong emphasis on data quality and security controls.

Qualifikationen

• 1+ years of data engineering/analytics engineering or backend roles building SQL-heavy data workloads in production.
• Strong SQL skills with window functions, CTEs, joins, and query tuning.

Aufgaben

• Design, write, and maintain Python jobs that execute SQL over Snowflake to produce transaction, settlement, fee, and reconciliation datasets.
• Develop performant, readable SQL using window functions, CTEs, and semi-structured data handling on large tables.
• Package workloads as Docker containers and run them on the existing scheduler.
• Ensure jobs are idempotent, runnable multiple times, parameterized, and observable with logging and alerting.

Kenntnisse

SQL

Python

Snowflake

Docker

Git

CI/CD

Unit tests

Data quality

Ausbildung

B.S. in CS/Engineering

Tools

Snowpark

Snowflake Python connector

pandas

Polars

dbt

Airflow

S3

Jobbeschreibung

Shift4 is seeking a hands-on Data Engineer to build reliable SQL and Python workloads in Snowflake for payments datasets and reporting.

Responsibilities

• Design, write, and maintain Python jobs that execute SQL over Snowflake to produce transaction, settlement, fee, and reconciliation datasets
• Develop performant, readable SQL using window functions, CTEs, incremental logic, and semi-structured (JSON/VARIANT) handling on large transaction tables
• Package workloads as Docker containers and run them on the existing scheduler
• Ensure jobs are idempotent, runnable multiple times, parameterized, and observable with logging and alerting for failures or bad outputs
• Implement data-quality checks and reconciliation controls to verify accuracy and consistent totals across sources and reporting periods
• Monitor Snowflake cost through warehouse sizing, clustering, query profiling, and avoiding wasteful scans
• Collaborate with finance, operations, and product stakeholders to translate payments questions into correct and maintainable queries
• Apply payments-appropriate security controls: secrets handling, least-privilege access, PII masking, and audit trails
• Document the job library and contribute to code review, testing, and CI practices
• Explore and apply AI tools to improve efficiency across query authoring, testing, and documentation

Requirements

• 1 to 3 years of experience in data engineering, analytics engineering, or backend roles building SQL-heavy data workloads in production
• Strong SQL skills including window functions, CTEs, aggregation and joins over large tables, query tuning, and reading execution plans
• Hands-on Snowflake experience: warehouses and cost awareness; stages and COPY INTO; streams and tasks; time travel; VARIANT/semi-structured data; Snowpark or the Snowflake Python connector
• Solid Python for data work: pandas or Polars, parameterized SQL, configuration and secrets management, structured logging, error handling, and unit tests
• Docker fundamentals: building images, running scheduled batch jobs in containers, and debugging container failures
• Git and basic CI/CD habits; comfort with code review and writing jobs that are testable and rerunnable
• Strong attention to numerical correctness, including investigating mismatched totals until the root cause is understood

Technologies

• Python, SQL, Snowflake, Docker
• pandas, Polars
• Git, Snowpark, Snowflake Python connector
• VARIANT, JSON, CTEs, window functions
• GitHub-style code review and CI/CD (generic)

Nice to Have

• Payments domain knowledge: authorization, capture, settlement, refunds, chargebacks, interchange and scheme fees, merchant and acquirer data models, reconciliation
• Orchestration and transformation tooling: Airflow, Prefect, Dagster, dbt
• AWS experience from the developer side: S3, IAM, Secrets Manager, ECS/Fargate scheduled tasks
• Data-quality frameworks and monitoring: Great Expectations, dbt tests, Soda, pipeline monitoring/alerting
• Experience in a PCI or other regulated environment
• Exposure to BI tools consuming outputs: Sigma, Tableau, Power BI, Looker

Location: Center Valley, PA (onsite)

Experience: 1+ years