Lead Streaming Data Engineer / Technical Lead

Compunnel, Inc.; Tampa (FL); On-site; USD 150,000 - 190,000; Full time

Job summary

Compunnel, Inc. is seeking a Lead Streaming Data Engineer / Technical Lead in Tampa, FL to design, build, and lead real‑time data processing platforms using Spark Structured Streaming, AWS Flink, and Kafka.

The role combines hands‑on development with architecture decisions and technical leadership. You will mentor onshore and offshore teams, drive design reviews and production readiness, optimize performance and cost, and partner with stakeholders to deliver enterprise‑scale streaming solutions

Qualifications

• 10+ years of overall professional experience.
• 5+ years of real-time streaming and distributed data processing experience.
• Hands-on expertise in Spark Structured Streaming.
• Hands-on experience with AWS Flink and Apache Kafka.
• Proficiency in Python or Java.
• Experience designing scalable, fault-tolerant streaming apps.
• Experience providing technical leadership and mentoring.
• Understanding security, governance, and compliance for enterprise data platforms.

Responsibilities

• Lead architecture and implementation of real-time data pipelines.
• Build scalable streaming apps using Spark, Flink, Kafka.
• Develop real-time pipelines with Spark and Python.
• Define standards, guidelines, and best practices.
• Drive design reviews, code reviews, and production readiness.
• Mentor onshore and offshore teams.
• Troubleshoot production incidents and ensure security/compliance.

Skills

Apache Spark Structured Streaming

AWS Flink

Apache Kafka

Python

Java

Tools

Python

Java

Job description

Lead Streaming Data Engineer / Technical Lead

Florida, Tampa

08/28/2026

Contract

Active

Job Description:

Job Summary

We are seeking an experienced Lead Streaming Data Engineer / Technical Lead to design, develop, and lead the implementation of large-scale real-time data processing platforms. The ideal candidate will have strong expertise in Apache Spark, AWS Flink, and Apache Kafka Streaming, along with proficiency in Python or Java. The role requires hands‑on experience developing Spark and Python data streaming pipelines, as well as technical leadership, architecture decision‑making, mentoring, and successful delivery of streaming data solutions.

Key Responsibilities

• Lead the architecture, design, and implementation of real‑time data processing pipelines.
• Build scalable and fault‑tolerant streaming applications using Apache Spark Structured Streaming, AWS Flink, and Apache Kafka.
• Develop and maintain real‑time streaming pipelines using Spark and Python.
• Apply distributed data processing principles to build reliable and scalable streaming solutions.
• Define technical standards, best practices, and coding guidelines for the team.
• Make architecture and technical design decisions for large‑scale streaming data solutions.
• Collaborate with business stakeholders, architects, product owners, and engineering teams to understand requirements and deliver effective solutions.
• Optimize streaming applications for performance, reliability, scalability, and cost efficiency.
• Drive solution design reviews, code reviews, and production readiness assessments.
• Mentor and guide development teams across onshore and offshore locations.
• Troubleshoot complex production issues and provide technical leadership during critical incidents.
• Ensure adherence to security, governance, and compliance requirements.
• Support the successful implementation and delivery of enterprise‑scale real‑time streaming and distributed data processing solutions.

Required Qualifications

• 10+ years of overall professional experience.
• 5+ years of experience working with real‑time streaming and distributed data processing solutions.
• Strong hands‑on experience with Apache Spark Structured Streaming.
• Strong hands‑on experience with AWS Flink.
• Strong hands‑on experience with Apache Kafka.
• Experience developing Spark and Python data streaming pipelines.
• Strong proficiency in Python or Java.
• Experience designing and building scalable and fault‑tolerant streaming applications.
• Strong understanding of real‑time data processing and distributed data processing architectures.
• Experience making architecture and technical design decisions.
• Experience providing technical leadership to engineering teams.
• Strong troubleshooting and problem‑solving skills in production environments.
• Experience mentoring and guiding development teams across onshore and offshore locations.
• Understanding of security, governance, and compliance requirements for enterprise data platforms.

Preferred Qualifications

• Strong experience optimizing streaming workloads for performance, reliability, scalability, and cost efficiency.
• Experience conducting solution design reviews, code reviews, and production readiness assessments.
• Experience working with cross‑functional teams including business stakeholders, architects, product owners, and engineering teams.