Lead Data Engineer_ Exp: 8+ Years (Azure+ Databricks + Datafacotry)

Atyeti Inc; Pune District; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Atyeti Inc is seeking a Senior Data Engineer to design and implement enterprise-scale data platforms on Azure. You will build scalable data pipelines, optimize Databricks notebooks, and ensure data quality across Bronze, Silver, and Gold layers.

With 8+ years in data engineering, you will work closely with stakeholders in an Agile environment, applying strong communication and problem-solving skills to deliver reliable data solutions.

Qualifications

• Bachelor's degree in Computer Science, IT, or related field.
• 8+ years in Data Engineering and Data Warehousing.
• 5+ years with Azure Databricks and PySpark.
• Experience in Agile/Scrum environments.
• Excellent communication and problem-solving skills.

Responsibilities

• Design, build, optimize enterprise-scale data pipelines on Azure.
• Develop ETL/ELT using Databricks, PySpark, and ADF.
• Maintain Delta Lake Bronze/Silver/Gold data layers.
• Collaborate with architects, analysts, and stakeholders to deliver data solutions.
• Ensure data quality, performance, and production support.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Azure Data Factory

Databricks Jobs & Workflows

Azure Data Lake Storage Gen2

Azure SQL Database / SQL Server

ETL/ELT Design & Development

Data Ingestion

Data Transformation

Data Modeling

Data Warehousing

Performance Tuning

CI/CD

Git

Data Quality

Monitoring

Production Support

Education

Bachelor's degree in Computer Science, Information Technology, or a related field

Tools

Azure Databricks

Azure Data Factory

Delta Lake

Azure Data Lake Storage Gen2

Azure SQL Database/SQL Server

Git

CI/CD pipelines

Job description

Job Overview : We are seeking an experienced Senior Data Engineer to design, build, optimize, and support enterprise-scale data platforms on Azure. The ideal candidate will possess strong expertise in data ingestion, transformation, orchestration, and data warehousing using technologies such as Azure Databricks, Azure Data Factory (ADF), Delta Lake, PySpark, and Azure SQL. The candidate will work closely with business stakeholders, data architects, analysts, and application development teams to deliver scalable, high-performance, and reliable data solutions.

Responsibilities:

• Design, develop, and maintain scalable data ingestion, transformation, and delivery pipelines.
• Build ETL/ELT solutions using Azure Databricks, PySpark, and Azure Data Factory (ADF).
• Develop and optimize Databricks notebooks, workflows, and scheduled jobs.
• Implement and maintain Delta Lake-based Bronze, Silver, and Gold data layers.
• Integrate data from databases, APIs, files, and cloud storage platforms.
• Ensure data quality, performance, reliability, and operational support for production pipelines.
• Collaborate with architects, analysts, and business stakeholders to deliver data solutions.
• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 8+ years of experience in Data Engineering and Data Warehousing.
• 5+ years of hands-on experience with Azure Databricks and PySpark.
• Experience working in Agile/Scrum environments.
• Excellent communication and problem-solving skills.

Skills:

Azure Databricks (PySpark, Spark SQL, Delta Lake), Azure Data Factory (ADF) ,Databricks Jobs & Workflows, Azure Data Lake, Storage (ADLS Gen2), Azure SQL Database /SQL Server, ETL/ELT Design & Development, Data Ingestion &Data Transformation, Data Modeling & Data Warehousing, Performance Tuning & Optimization, CI/CD, Git, Data Quality, Monitoring, and Production Support