AI-Driven Data Pipeline Architect

Cognizant; Dubai; On-site; AED 320,000 - 520,000; Full time

Job summary

Cognizant in Dubai is seeking an experienced data engineer to design, build, and maintain batch and real-time data pipelines. You will develop data products using Databricks Lakeflow Declarative Pipelines, PySpark, and SQL, and build streaming solutions with Spark Structured Streaming.

Responsibilities include tuning for performance, CI/CD, and production support. Exposure to AI-assisted development tools and collaboration in Agile environments is favored.

Qualifications

• : Design, build and maintain batch and real-time data pipelines.
• : Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
• : Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines
• : Optimise pipelines for scalability, reliability and performance.
• : Develop automated tests and participate in code reviews.
• : Contribute to CI/CD and engineering best practices.
• : Monitor, troubleshoot and support production workloads.
• : Use AI-assisted development tools to improve productivity while maintaining high engineering standards.

Responsibilities

• : Design, build and maintain batch and real-time data pipelines.
• : Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
• : Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines
• : Optimise pipelines for scalability, reliability and performance.
• : Develop automated tests and participate in code reviews.
• : Contribute to CI/CD and engineering best practices.
• : Monitor, troubleshoot and support production workloads.
• : Use AI-assisted development tools to improve productivity while maintaining high engineering standards.

Skills

Databricks

Lakeflow Declarative Pipelines

PySpark

SQL

Spark Structured Streaming

Auto Loader

Unity Catalog

Python

Delta Lake

Azure Event Hubs

Git

Unit testing

CI/CD

Analytical skills

DB Genie

MCP Server

Azure DevOps

GitHub Copilot

Tools

Azure DevOps

GitHub Copilot

Job description

Cognizant in Dubai is seeking an experienced data engineer to design, build, and maintain batch and real-time data pipelines. You will develop data products using Databricks Lakeflow Declarative Pipelines, PySpark, and SQL, and build streaming solutions with Spark Structured Streaming.

Responsibilities include tuning for performance, CI/CD, and production support. Exposure to AI-assisted development tools and collaboration in Agile environments is favored.