Data Engineer - Spark/Databricks, Cloud Data Pipelines

Virtusa; Dubai; On-site; AED 330,000 - 480,000; Full time

Job summary

Virtusa is seeking a Data Bricks Engineer to design and implement data pipelines for banking applications. The role focuses on building scalable Spark/Databricks pipelines, streaming data ingestion with Kafka, and cloud-based data storage and orchestration.

The ideal candidate will have strong SQL, Python/Scala/Java skills and experience with Delta Lake, data modelling, and CI/CD workflows. Collaboration with cross-functional teams is essential to success.

Qualifications

• 58 years of experience designing and building data pipelines with Spark/Databricks or equivalent big data frameworks.
• Hands-on expertise with streaming and messaging systems such as Kafka (pub-sub) and Confluent Cloud.
• Experience creating producers/consumers and topics and integrating them into downstream processing.
• Deep understanding of relational databases and CDC; proficiency in SQL Server/Oracle or other RDBMS.
• Proficiency in Python, Scala or Java and solid SQL for data manipulation.
• Cloud platform experience with Azure or AWS services for storage, compute and orchestration.
• Data modelling and warehousing knowledge, Delta Lake and Lakehouse concepts.
• Version control, Git and CI/CD pipelines; automating deployments and IaC.

Responsibilities

• Design and build robust data pipelines using Databricks/Spark for banking data.
• Implement streaming architectures and ensure reliable data ingestion from sources like Kafka.
• Model and transform data for downstream analytics and reporting.
• Collaborate with cross-functional teams and communicate complex tech concepts clearly.

Skills

Databricks

Spark

Kafka

SQL

Python/Scala/Java

Cloud (Azure/AWS)

Delta Lake

CDC/Change Data Capture

Data Modeling

Streaming architectures

Tools

NiFi

Flume

Git

CI/CD

Job description

Virtusa is seeking a Data Bricks Engineer to design and implement data pipelines for banking applications. The role focuses on building scalable Spark/Databricks pipelines, streaming data ingestion with Kafka, and cloud-based data storage and orchestration.

The ideal candidate will have strong SQL, Python/Scala/Java skills and experience with Delta Lake, data modelling, and CI/CD workflows. Collaboration with cross-functional teams is essential to success.