Data Architect Engineer

Adani Group; Ahmedabad District; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

Adani Group is seeking a data engineer to design, develop, and optimize data pipelines on Microsoft Azure, focusing on Azure Databricks and Delta Lake. You will transform raw enterprise data into analytics-ready datasets powering reporting, BI, and AI initiatives.

You will collaborate with data scientists, analysts, and software engineers, implement medallion architecture, and ensure secure data access with ADLS Gen2, AAD, and Key Vault while aligning with governance standards.

Qualifications

• Bachelor's and/or master’s degree in computer science, IT Management, Information Technology, Engineering.
• Math / Statistics background is added advantage.
• Good communications and listening skills are critical components of this job.

Responsibilities

• Design, develop, and optimize Azure Lakehouse architectures with ADLS Gen2 and Delta Lake.
• Build scalable batch and streaming data pipelines using Databricks and PySpark.
• Integrate Databricks with Azure Data Factory, Synapse, and BI tools; ensure data governance and security.
• Collaborate with analysts, data scientists, and stakeholders; document designs and guidelines.

Skills

Delta Lake

Lakehouse architecture

Azure Databricks

ADLS Gen2

Education

Bachelor's or Master’s degree in Computer Science/IT/Engineering

Tools

PySpark

Spark SQL

Azure Data Factory

Job description

Purpose/Objective

• In this role, you will be instrumental in designing, developing, and optimizing data pipelines on the Microsoft Azure cloud platform, with a primary focus on Azure Databricks and Delta Lake.
• You will work closely with cross-functional teams including Data Scientists, Analysts, and Software Engineers to deliver high-quality, reliable data solutions that drive business intelligence and machine learning initiatives.
• The ideal candidate will play a key role in transforming raw enterprise data into trusted, analytics ready datasets that power reporting, advanced analytics, and AI/ML initiatives

Key Responsibilities of Role

• - Data Engineering & Lakehouse Design- Design, implement, and optimize Azure Lakehouse architectures using Azure Data Lake Storage (ADLS Gen2) and Databricks (Delta Lake) - Implement medallion architecture (Bronze, Silver, Gold layers) for structured and unstructured data - Develop scalable batch and streaming data pipelines using Databricks - Databricks Development- Build and maintain Databricks notebooks using PySpark / Spark SQL - Optimize Spark jobs for performance and cost (partitioning, caching, Delta optimizations) - Implement Delta Lake features: ACID transactions, schema evolution, time travel, and data versioning - Data Integration & Processing Develop, test, and deploy highly scalable ETL/ELT data pipelines using Azure Databricks (PySpark/SQL) and Azure Data Factory (ADF).
• - Develop ingestion pipelines from multiple sources: - Relational databases - APIs - Files (CSV, JSON, Parquet) - Streaming sources (Event Hubs / Kafka, if applicable) - Handle incremental loads, CDC, and data quality validations - Azure Ecosystem Integration- Integrate Databricks with: - Azure Data Factory / Synapse Pipelines - Azure Functions & Logic Apps (where applicable) - Power BI and other BI tools - Implement secure access using Azure Active Directory, Key Vault, Managed Identities - Data Governance & Quality- Apply data validation, reconciliation, and quality checks - Support data cataloging, lineage, and governance (Unity Catalog preferred) - Ensure compliance with security and regulatory standards - Collaboration & Delivery- Work closely with business analysts, data scientists, and stakeholders - Participate in solution design, estimation, and code reviews - Contribute to documentation, best practices, and reusable frameworks -- Documenting architectural designs, configurations, and implementation guidelines for reference and knowledge sharing.

Technical Competencies

• Experience with Delta Lake and Lakehouse architecture
• Strong experience with Azure Databricks
• Proficiency with Azure Data Lake Storage (ADLS Gen2)
• Experience with Azure Data Factory or equivalent orchestration tools

Qualifications and Experience

• - Bachelor's and/or master’s degree in computer science, IT Management, Information Technology, Engineering.
• (Math / Statistics background is added advantage).
• - Good communications and listening skills are critical components of this job.
• Must be able to interact with development and infrastructure teams, business and vendors including the training and mentoring of team members.
• - A proven track record for the successful delivery of cloud data warehouse solutions and related components