Lead Dataflow Platform Engineer

Socket.dev; Atlanta (GA); On-site; USD 127,000 - 149,000; Full time

Benefits offered by this job

Healthcare (medical, dental, vision)

401(k) and employer-funded retirement

Paid vacation and holidays

Parental leave

Job summary

U.S. Bank is seeking a Software Engineer to contribute to our technology initiatives in our digital transformation journey. You will own the Dataflow platform end-to-end, from UI to DAG generation and Spark/Flink execution on Azure Kubernetes Service.

You will lead a small team of engineers, mentor peers, and collaborate across platform teams to deliver reliable data pipelines and improve operational excellence in a fast-paced environment.

Qualifications

• Bachelor’s degree or equivalent work experience.
• Six to eight years of relevant experience.
• Experience in data engineering and cloud platforms.

Responsibilities

• Architect and evolve the Dataflow platform (Airflow 3.x, SparkKubernetesOperator, FlinkKubernetesOperator).
• Lead a team of 3-5 engineers; mentor and set technical direction.
• Design and implement CDC streaming pipelines (Flink) and batch ELT pipelines (Spark).
• Own Kubernetes deployment patterns — Helm charts, pod templates, resource tuning on AKS.
• Build and maintain the self-service UI (React/Vite) and API server (Node.js/Express).
• Drive reliability: monitoring, alerting, SLA enforcement, auto-retry, and incident response.
• Collaborate with platform teams on networking (Istio egress), secrets (Vault), storage (ADLS Gen2).
• Manage CI/CD pipelines (GitLab) and container image builds for Airflow/Spark/Flink.
• Define data quality frameworks (Great Expectations, custom DQ rules).

Skills

Apache Airflow

Apache Spark

Apache Flink

Kubernetes

Azure

Python

SQL

JavaScript/TypeScript

GitLab CI/CD

Communication

Education

Bachelor’s degree or equivalent

Tools

React

Node.js

AKS

Istio

Vault

ADLS Gen2

GitLab

Job description

U.S. Bank is seeking a Software Engineer to contribute to our technology initiatives in our digital transformation journey. You will own the Dataflow platform end-to-end, from UI to DAG generation and Spark/Flink execution on Azure Kubernetes Service.

You will lead a small team of engineers, mentor peers, and collaborate across platform teams to deliver reliable data pipelines and improve operational excellence in a fast-paced environment.