Lead Real-Time Data Pipeline Engineer

Dale Workforce Solutions; Saint Paul (MN); On-site; USD 120,000 - 160,000; Full time

Job summary

Dale Workforce Solutions is seeking a software engineer to design and manage real-time streaming pipelines using Apache Kafka and Apache Flink. You will integrate diverse data sources with AWS and Snowflake, ensure data quality, and drive reliability across platforms.

The role emphasizes automation with Terraform and CI/CD, performance tuning of pipelines, and providing hands-on Kafka guidance to teams. Ideal candidates have 3+ years in Kafka and distributed systems and 2+ years in Python.

Qualifications

• Requires at least three years of software development experience focused on Apache Kafka and distributed systems.
• Includes at least two years of Python experience.
• Hands-on experience with Confluent Kafka, Schema Registry, Terraform, and AWS.

Responsibilities

• Develop and manage real-time streaming pipelines using Apache Kafka and Apache Flink.
• Integrate data sources with AWS and Snowflake while maintaining data quality.
• Automate infrastructure and deployments with Terraform and CI/CD.

Skills

Apache Kafka

Confluent Cloud

Python

Distributed Systems

Apache Flink

Terraform

AWS

Schema Registry

Data Integration

ETL

Data Pipeline Orchestration

CI/CD

Operational Monitoring

Snowflake

Java

Kafka Connect

Job description

Dale Workforce Solutions is seeking a software engineer to design and manage real-time streaming pipelines using Apache Kafka and Apache Flink. You will integrate diverse data sources with AWS and Snowflake, ensure data quality, and drive reliability across platforms.

The role emphasizes automation with Terraform and CI/CD, performance tuning of pipelines, and providing hands-on Kafka guidance to teams. Ideal candidates have 3+ years in Kafka and distributed systems and 2+ years in Python.