Lead Data Engineer

Azilen Technologies; Ahmedabad District; On-site; INR 4,500,000 - 10,000,000; Full time

Job summary

Azilen Technologies in India is seeking an experienced Data Engineering Lead to design, build, and optimize scalable data platforms for global clients. You will lead modern data pipelines, cloud-native architectures, and data infrastructure while collaborating with cross-functional teams to deliver robust data solutions.

The role requires 8+ years in data engineering with leadership experience, strong Python, SQL, PySpark/Scala, and hands-on cloud proficiency (AWS/Azure/GCP).

Qualifications

• 8+ years of data engineering experience with leadership exposure.
• Strong Python, SQL, and PySpark/Scala for large-scale processing.
• Hands-on cloud data ecosystem knowledge (AWS/Azure/GCP).
• Experience with modern data platforms and data governance.

Responsibilities

• Lead design and implementation of scalable data pipelines and architectures.
• Collaborate with cross-functional teams and mentor engineers.
• Drive data governance, quality, and security practices.
• Oversee multiple client engagements and ensure timely delivery.

Skills

Python

SQL

PySpark/Scala

Cloud platforms (AWS/Azure/GCP)

Data pipelines

ETL/ELT design

Airflow

DevOps / IaC

Leadership

Mentoring

Stakeholder management

Communication

Tools

Docker

Kubernetes

Terraform

GitHub Actions

Azure DevOps

Jenkins

Snowflake

Databricks

Delta Lake

Kafka

Spark Streaming

Job description

Design, build, and optimize scalable data engineering solutions that enable reliable, high-performance data platforms. Lead the development of modern data pipelines, cloud-native architectures, and data infrastructure while collaborating with cross-functional teams to deliver robust, client-focused data solutions.

Who You Are:

• 8+ years of experience in Data Engineering, with at least 2+ years in a Technical Lead or supervisory role
• Prior experience working in software services, IT consulting, or agency environments, managing multiple client engagements
• Strong proficiency in Python, SQL, and PySpark or Scala for large-scale data processing
• Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its data ecosystem (e.g., AWS Glue, Redshift, EMR, Azure Data Factory, Synapse, GCP BigQuery)
• Advanced experience working with modern data platforms such as Snowflake, Databricks, or Delta Lake
• Strong expertise in designing and building scalable ETL/ELT pipelines and data transformation workflows
• Experience with workflow orchestration tools such as Apache Airflow, dbt, or Prefect
• Hands-on experience with real-time data streaming technologies like Apache Kafka or Spark Streaming
• Good understanding of software engineering best practices, version control, testing, and CI/CD processes
• Excellent communication and stakeholder management skills with the ability to lead technical discussions and mentor engineering teams

What Will Excite Us:

• Experience designing enterprise-scale data platforms and modern cloud data architectures
• Strong expertise in Databricks, Snowflake, or Delta Lake implementations
• Experience building real-time data pipelines using Kafka, Spark Streaming, or similar technologies
• Hands-on experience with DevOps and Infrastructure-as-Code tools such as Docker, Kubernetes, Terraform, GitHub Actions, Azure DevOps, or Jenkins
• Experience optimizing data pipeline performance, scalability, reliability, and cost efficiency
• Strong understanding of data governance, data quality, security, and performance optimization
• Experience leading technical teams, conducting code reviews, and mentoring data engineers
• Exposure to modern DataOps practices and automation frameworks
• Ability to work across multiple client projects while ensuring high-quality delivery

What Will Excite You:

• Opportunity to architect and build large-scale cloud-native data platforms for global clients
• Work with modern technologies including Databricks, Snowflake, Kafka, Airflow, and leading cloud ecosystems
• Lead technical initiatives while mentoring and growing high-performing engineering teams
• Solve complex data engineering challenges involving large-scale data processing and real-time analytics
• Collaborate with cross-functional teams in a fast-paced, innovation-driven environment
• Competitive compensation, continuous learning opportunities, and significant career growth