Senior Streaming Data Engineer - Databricks & Kafka

Virtusa; Dubai; On-site; AED 250,000 - 320,000; Full time

Job summary

A leading technology company in Dubai is seeking a Senior Software Engineer to build scalable data ingestion and streaming platforms. The candidate will design and develop streaming ingestion pipelines, ensuring quality and fault tolerance in data processing. Ideal applicants should have 5-8 years experience with Apache Spark or similar frameworks, strong SQL skills, and expertise in streaming systems like Kafka. This role offers a chance to work on innovative data solutions within a collaborative team environment.

Qualifications

• 5–8 years of experience designing and building data pipelines using Apache Spark or similar frameworks.
• Hands-on expertise with streaming systems like Apache Kafka, Confluent Cloud, or RabbitMQ.
• Deep understanding of relational databases and change data capture (CDC).
• Proficiency in programming languages such as Python, Scala, or Java.

Responsibilities

• Design and develop streaming ingestion pipelines.
• Implement change data capture and deduplication logic.
• Ensure data quality and fault tolerance.
• Collaborate on data platform architecture.

Skills

Kafka

Confluent

Apache Spark

Data Lake Integration

Real-Time Data Ingestion

Python

Scala

Java

SQL

Tools

Databricks

Terraform

Jenkins

Airflow

Job description

A leading technology company in Dubai is seeking a Senior Software Engineer to build scalable data ingestion and streaming platforms. The candidate will design and develop streaming ingestion pipelines, ensuring quality and fault tolerance in data processing. Ideal applicants should have 5-8 years experience with Apache Spark or similar frameworks, strong SQL skills, and expertise in streaming systems like Kafka. This role offers a chance to work on innovative data solutions within a collaborative team environment.