Senior Data & AI Engineer (Databricks Specialist)

Cognida; Hyderabad; Hybride; INR 2 500 000 - 4 500 000; Plein temps

Résumé du poste

Cognida in Hyderabad, India, seeks a Senior Data & AI Engineer (Databricks Specialist) with 8+ years of experience to lead data pipeline design on the Databricks Lakehouse. You will own data governance using Unity Catalog, ensure secure access, and drive data lineage across platforms.

You will rapidly prototype end-to-end data and AI demos, build robust ETL/ELT pipelines with Spark/Delta Lake, and optimize performance and costs. You will collaborate globally with minimal supervision.

Qualifications

• Proven experience implementing Unity Catalog for governance and data lineage.
• Deep knowledge of Databricks architecture, Delta Lake, and serverless compute.
• Strong Python/PySpark skills and SQL proficiency.
• Experience building real-time and batch ETL/ELT pipelines.

Responsabilités

• Design and implement Databricks-based data pipelines with governance.
• Prototype end-to-end data and AI demos for stakeholders and clients.
• Develop scalable ETL/ELT pipelines using Spark and Delta Lake.
• Collaborate autonomously with global, cross-functional teams.

Connaissances

Unity Catalog mastery

Python/PySpark

SQL

Delta Lake

Databricks platform

Spark / Structured streaming

LLM integration

MLflow

POC development

Outils

Databricks platform

Delta Lake

MLflow

Azure/AWS/GCP

Description du poste

Position : Senior Data & AI Engineer (Databricks Specialist)

Experience : 8+ years

Location : Hyderabad, India

Key Responsibilities

Databricks Architecture & Governance

Design and implement robust data pipelines utilizing the Databricks Lakehouse platform. Take ownership of data governance, security, and fine-grained lineage using Unity Catalog.

Rapid Prototyping & Demos

Quickly conceptualize, build, and present functional, end-to-end data and AI demos (e.g., live dashboards, real-time pipelines, or LLM/RAG prototypes) to stakeholders and clients.

Core Data Engineering

Build scalable, efficient ETL/ELT pipelines using Spark (PySpark/SQL), Delta Lake, and structured streaming. Ensure optimal performance tuning, partitioning, and cost-management of Databricks clusters.

Global Collaboration & Autonomy

Work independently with minimal supervision, managing your own schedule to align with global, cross-functional teams.

Required Skills & Qualifications

Databricks & Governance Mastery

Proven track record implementing Databricks Unity Catalog for data governance, access control, and data lineage.

Deep understanding of Databricks architecture, including Delta Lake, serverless compute, and Workflows.

Core Data Engineering Excellence

Strong foundational knowledge of data engineering principles (star schemas, dimensional modeling, data warehousing vs. data lakes).

Advanced proficiency in Python/PySpark and SQL.

Experience building robust orchestrations and handling both batch and real-time streaming data.

Experience in building AI/ML workloads (e.g., MLflow, LLM integration, or RAG frameworks) within Databricks.

Agility & Communication

Exceptional ability to build quick, concrete POCs (Proof of Concepts) and technical demos that clearly translate data capabilities into business solutions.

Excellent communication skills, with the ability to clearly articulate technical concepts to both technical and non-technical audiences.

Preferred Qualifications

Certifications such as Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.

Experience integrating Databricks with enterprise ecosystems, cloud provider native services (AWS/Azure/GCP), or modern BI tools.