AI Architect

Cognizant; Dubai; On-site; AED 320,000 - 520,000; Full time

Job summary

Cognizant in Dubai is seeking an experienced data engineer to design, build, and maintain batch and real-time data pipelines. You will develop data products using Databricks Lakeflow Declarative Pipelines, PySpark, and SQL, and build streaming solutions with Spark Structured Streaming.

Responsibilities include tuning for performance, CI/CD, and production support. Exposure to AI-assisted development tools and collaboration in Agile environments is favored.

Qualifications

• : Design, build and maintain batch and real-time data pipelines.
• : Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
• : Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines
• : Optimise pipelines for scalability, reliability and performance.
• : Develop automated tests and participate in code reviews.
• : Contribute to CI/CD and engineering best practices.
• : Monitor, troubleshoot and support production workloads.
• : Use AI-assisted development tools to improve productivity while maintaining high engineering standards.

Responsibilities

• : Design, build and maintain batch and real-time data pipelines.
• : Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
• : Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines
• : Optimise pipelines for scalability, reliability and performance.
• : Develop automated tests and participate in code reviews.
• : Contribute to CI/CD and engineering best practices.
• : Monitor, troubleshoot and support production workloads.
• : Use AI-assisted development tools to improve productivity while maintaining high engineering standards.

Skills

Databricks

Lakeflow Declarative Pipelines

PySpark

SQL

Spark Structured Streaming

Auto Loader

Unity Catalog

Python

Delta Lake

Azure Event Hubs

Git

Unit testing

CI/CD

Analytical skills

DB Genie

MCP Server

Azure DevOps

GitHub Copilot

Tools

Azure DevOps

GitHub Copilot

Job description

Key Responsibilities

• Design, build and maintain batch and real-time data pipelines.
  


• Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
  


• Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines
  


• Optimise pipelines for scalability, reliability and performance.
  


• Develop automated tests and participate in code reviews.
  


• Contribute to CI/CD and engineering best practices.
  


• Monitor, troubleshoot and support production workloads.
  


• Use AI-assisted development tools to improve productivity while maintaining high engineering standards.
  

Essential Skills

• Databricks
  


• Lakeflow Declarative Pipelines and DXQ, DB Genie, MCP Server
  


• Apache Spark (PySpark)
  


• Spark Structured Streaming
  


• Auto Loader
  


• Unity Catalog
  


• Python
  


• SQL
  


• Delta Lake
  


• Azure Event Hubs or equivalent event streaming platforms
  


• Git and collaborative development workflows
  


• Unit testing
  


• CI/CD
  


• Strong analytical and problem-solving skills
  

Nice to Have

• Performance tuning and Spark optimisation
  


• Data modelling
  


• Azure DevOps
  


• AI coding assistants (GitHub Copilot, Cursor or similar)
  


• Experience working in Agile or SAFe environments
  

We're looking for an engineer who:

• Takes ownership from design through production support.
  


• Writes clean, maintainable and well-tested code.
  


• Builds reusable, production-ready solutions.
  


• Collaborates effectively with teammates and stakeholders.
  


• Continuously learns and embraces modern engineering practices.