Senior Stream Processing Framework Specialist

Small Cap Equity Advisors - 30 Yrs.; Abu Dhabi; Remote; AED 350,000 - 600,000; Full time

Benefits offered by this job

100% remote

Flexible schedule

Professional development

Collaborative team

Job summary

Small Cap Equity Advisors - 30 Yrs. seeks a Senior Stream Processing Framework Specialist to design, optimize, and maintain real-time data processing systems. You will develop scalable streaming pipelines and ensure reliable data delivery across distributed platforms.

The role focuses on Flink, Kafka Streams, Spark Structured Streaming, and related frameworks, with a strong emphasis on latency, throughput, and fault-tolerance. Remote work from the UAE is offered.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Software Engineering, or a related field.
• 5–8 years of experience in data engineering, distributed systems, or real-time stream processing.
• Strong knowledge of streaming architectures, event-driven systems, and distributed data processing.
• Proficiency in Java, Scala, Python, or another relevant programming language.
• Understanding of Kafka, event-time semantics, stateful processing, and fault tolerance.
• Familiarity with cloud platforms, Docker, Kubernetes, and CI/CD pipelines.
• Experience with data monitoring, pipeline testing, and performance tuning.
• Strong analytical, troubleshooting, and communication skills.
• Ability to work independently in a fully remote environment.

Responsibilities

• Design and implement real-time data streaming pipelines and processing architectures.
• Develop and optimize stream processing applications using Apache Flink, Kafka Streams, Spark Structured Streaming, or similar frameworks.
• Monitor processing latency, throughput, data accuracy, and resource utilization.
• Implement event-time processing, windowing, state management, and fault-tolerance mechanisms.
• Troubleshoot data delays, processing failures, duplicate events, and pipeline bottlenecks.
• Integrate streaming platforms with databases, data lakes, APIs, and cloud services.
• Optimize checkpointing, state recovery, parallelism, and resource allocation.
• Establish data quality controls, monitoring, alerting, and automated recovery procedures.
• Collaborate with data engineering, software development, and DevOps teams to improve streaming reliability.
• Document architecture decisions, operational procedures, and performance optimization standards.

Skills

Streaming pipelines

Event-time semantics

Distributed systems

Java/Scala/Python

Cloud & containers

CI/CD

Troubleshooting

Remote work

Education

Bachelor's degree in CS/SE

Tools

Apache Flink

Kafka Streams

Spark Structured Streaming

Docker

Kubernetes

CI/CD pipelines

Job description

Job Title: Senior Stream Processing Framework Specialist

Location: United Arab Emirates (Remote)

Employment Type: Full-Time

Work Arrangement: 100% Remote — Work From Home

About the Role

We are seeking an experienced Senior Stream Processing Framework Specialist to design, optimize, and maintain real-time data processing systems. The successful candidate will develop scalable streaming pipelines, improve processing performance, and ensure reliable data delivery across distributed platforms.

Key Responsibilities

• Design and implement real-time data streaming pipelines and processing architectures.
• Develop and optimize stream processing applications using Apache Flink, Kafka Streams, Spark Structured Streaming, or similar frameworks.
• Monitor processing latency, throughput, data accuracy, and resource utilization.
• Implement event-time processing, windowing, state management, and fault-tolerance mechanisms.
• Troubleshoot data delays, processing failures, duplicate events, and pipeline bottlenecks.
• Integrate streaming platforms with databases, data lakes, APIs, and cloud services.
• Optimize checkpointing, state recovery, parallelism, and resource allocation.
• Establish data quality controls, monitoring, alerting, and automated recovery procedures.
• Collaborate with data engineering, software development, and DevOps teams to improve streaming reliability.
• Document architecture decisions, operational procedures, and performance optimization standards.

Qualifications and Experience

• Bachelor's degree in Computer Science, Data Engineering, Software Engineering, or a related field.
• 5–8 years of experience in data engineering, distributed systems, or real-time stream processing.
• Strong knowledge of streaming architectures, event-driven systems, and distributed data processing.
• Practical experience with Apache Flink, Kafka Streams, Spark Structured Streaming, or equivalent frameworks.
• Proficiency in Java, Scala, Python, or another relevant programming language.
• Understanding of Kafka, event-time semantics, stateful processing, and fault tolerance.
• Familiarity with cloud platforms, Docker, Kubernetes, and CI/CD pipelines.
• Experience with data monitoring, pipeline testing, and performance tuning.
• Strong analytical, troubleshooting, and communication skills.
• Ability to work independently in a fully remote environment.

What We Offer

• Competitive salary based on experience and qualifications.
• 100% remote work with no routine office attendance.
• Professional development and career advancement opportunities.
• Collaborative environment focused on real-time data innovation and engineering excellence.