Data Engineer - Lead

Iris Software; Dadri; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

A leading technology company in India is seeking a Lead Data Engineer responsible for architecting enterprise-grade data platforms. The ideal candidate will have over 8 years of experience in data engineering, strong expertise in Python and PySpark, and experience with AWS and Databricks. Key responsibilities include leading engineering teams, collaborating on AI/ML initiatives, and ensuring data governance. This role offers a strategic opportunity to drive innovation in data architecture and analytics.

Qualifications

• 8+ years of experience in data engineering and architecture with proven leadership.
• Deep expertise in Python and PySpark.
• Experience with visualizing data governance in regulated industries.

Responsibilities

• Architect scalable data platforms for enterprise analytics and AI/ML initiatives.
• Lead engineering teams providing mentorship and technical leadership.
• Collaborate on operationalizing models and implementing AI orchestration frameworks.

Skills

Python

PySpark

AWS

Databricks

Data Governance

AI/ML

Cloud Solutions

Education

Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or related field

Tools

Databricks

CI/CD

Job description

Overview

The Lead Data Engineer is a strategic and technical leadership role responsible for architecting, scaling, and evolving enterprise-grade data platforms that enable advanced analytics, AI/ML, and data-driven decision-making. Reporting to the Senior Director of Data Platforms, this role will lead the design and governance of modern data architectures, drive innovation in AI orchestration, and ensure the delivery of secure, compliant, and high-performing data solutions. This position combines hands-on engineering expertise with architectural vision and cross-functional leadership. The Lead Data Architect will guide engineering teams, influence platform strategy, and establish best practices across the organization’s data ecosystem.

Basic Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or related field
• 8+ years of experience in data engineering and architecture, with a proven track record of leading large-scale data initiatives
• Deep expertise in Python, PySpark
• Strong hands-on experience with Databricks (Spark, Delta Lake, Workflows); strong experience with AWS (S3, IAM, Textract, Bedrock or equivalent); demonstrated success in architecting and deploying AI/ML pipelines in production
• Experience with design and implementation of scalable document ingestion pipelines using Databricks Auto Loader and AWS S3
• Understanding of vector embeddings and semantic search
• Strong understanding of data governance, privacy, and compliance in regulated industries (healthcare, life sciences)

Preferred Qualifications

• Advanced knowledge of data modeling, lakehouse/lake/warehouse design, and performance optimization
• Familiarity with generative AI platforms and use cases
• Contributions to open-source projects or thought leadership in data engineering/architecture
• Experience with Agile methodologies, CI/CD, and DevOps practices
• Exposure to FastAPI, or API-based ML services; experience evaluating LLM output quality

Key Responsibilities

• Architect scalable data platforms: Design and oversee implementation of robust data lakehouse, data mesh, and real-time streaming architectures to support enterprise analytics and AI/ML initiatives
• Lead engineering teams: Provide technical leadership and mentorship to data engineers, fostering a culture of excellence, innovation, and continuous improvement
• AI/ML enablement: Collaborate with Data Science and ML Engineering teams to operationalize models, implement AI orchestration frameworks (e.g., MLflow, Airflow), and ensure scalable deployment pipelines
• Platform strategy & governance: Define and enforce architectural standards, data governance policies, and compliance frameworks (HIPAA, SOC 2, GDPR, etc.) across the data platform
• Performance & reliability optimization: Drive observability, automation, and performance tuning across data pipelines and infrastructure to ensure reliability at scale
• Cross-functional collaboration: Partner with product, analytics, compliance, and infrastructure teams to align data architecture with business goals and regulatory requirements
• Innovation & thought leadership: Stay ahead of industry trends, evaluate emerging technologies, and contribute to strategic decisions on platform evolution, including generative AI integration and event-driven systems

Mandatory Competencies

• Cloud - AWS (S3, S3 Glacier, EBS)
• Development Tools and Management - CI/CD
• Communication
• Big Data - PySpark
• Data Science and Machine Learning - Python
• Data Science and Machine Learning - AI/ML