Data Engineer - Lead

Iris Software, Inc.; Dadri; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

Iris Software, Inc. is seeking a Lead Data Engineer to architect, scale, and govern enterprise-grade data platforms enabling advanced analytics, AI/ML, and data-driven decisions.

You will lead teams, drive AI orchestration, and ensure secure, compliant, high-performing data solutions across the organization.

Qualifications

• Bachelor's or Master's degree in CS/Engineering/Data Science or related field.
• 8+ years of data engineering and architecture experience with large-scale initiatives.
• Deep expertise in Python and PySpark, with Databricks experience.
• Strong AWS experience (S3, IAM, Textract, Bedrock or equivalent).
• Experience with scalable document ingestion pipelines using Databricks Auto Loader and AWS S3.
• Knowledge of vector embeddings, semantic search, and data governance concepts.

Responsibilities

• Lead engineering teams, providing technical leadership and mentorship.
• Collaborate with Data Science and ML teams to operationalize models and AI orchestration.
• Define architectural standards and enforce governance and regulatory compliance.
• Drive observability, automation, and performance improvements across pipelines.
• Partner with product, analytics, and infra teams to align data architecture with goals.
• Stay ahead of trends and guide platform evolution including AI integration.

Skills

Python

PySpark

Leadership

Cross-functional collaboration

Data governance

AI/ML enablement

Education

Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or related field

Tools

Databricks

AWS

Spark / Delta Lake

MLflow / Airflow

FastAPI (API-based ML services)

Job description

Select how often (in days) to receive an alert: Create Alert

Why Join Iris?
Are you ready to do the best work of your career at one ofIndia's Top 25 Best Workplaces in IT industry? Do you want to grow in an award-winning culture thattruly values your talent and ambitions?
Join Iris Software — one offastest-growing IT services companies— whereyou own and shape your success story.

About Us
At Iris Software, our vision is to be our client's most trusted technology partner, and the first choice for the industry's top professionals to realize their full potential.

With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transportation & logistics, and professional services.

Our work covers complex, mission-critical applications with the latest technologies, such as high-value complex Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation.

Working with Us
At Iris, every role is more than a job — it's a launchpad for growth.

Our Employee Value Proposition, "Build Your Future. Own Your Journey.">reflects our belief that people thrive when they have ownership of their career and the right opportunities to shape it.

We foster a culture where your potential is valued, your voice matters, and your work creates real impact. With cutting-edge projects, personalized career development, continuous learning and mentorship, we support you to grow and become your best — both personally and professionally.

Curious what it's like to work at Iris? Head to this video for an inside look at the people, the passion, and the possibilities. Watch it here .

Job Description

Job Overview Summary

• The Lead Data Engineer is a strategic and technical leadership role responsible for architecting, scaling, and evolving enterprise-grade data platforms that enable advanced analytics, AI/ML, and data-driven decision-making. Reporting to the Senior Director of Data Platforms, this role will lead the design and governance of modern data architectures, drive innovation in AI orchestration, and ensure the delivery of secure, compliant, and high-performing data solutions.
• This position combines hands-on engineering expertise with architectural vision and cross-functional leadership. The Lead Data Engineer will guide engineering teams, influence platform strategy, and establish best practices across the organization’s data ecosystem.

Basic Qualifications :

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or related field.
• 8+ years of experience in data engineering and architecture, with a proven track record of leading large-scale data initiatives.
• Deep expertise in Python, PySpark.
• Strong hands-on experience with Databricks (Spark, Delta Lake, Workflows)
• Strong experience with AWS (S3, IAM, Textract, Bedrock or equivalent)
• Experience with design and implement scalable document ingestion pipelines using Databricks Auto Loader and AWS S3.
• Understanding of vector embeddings and semantic search
• Strong understanding of data governance, privacy, and compliance in regulated industries (healthcare, life sciences).

Good to Have :

• Advanced knowledge of data modeling, lakehouse/lake/warehouse design, and performance optimization.
• Familiarity with generative AI platforms and use cases.
• Contributions to open-source projects or thought leadership in data engineering/architecture.
• Experience with Agile methodologies, CI/CD, and DevOps practices.
• Exposure to FastAPI, or API-based ML services

Key Responsibilities :

• Lead Engineering Teams: Provide technical leadership and mentorship to data engineers, fostering a culture of excellence, innovation, and continuous improvement.
• AI/ML Enablement: Collaborate with Data Science and ML Engineering teams to operationalize models, implement AI orchestration frameworks (e.g., MLflow, Airflow), and ensure scalable deployment pipelines.
• Platform Strategy & Governance: Define and enforce architectural standards, data governance policies, and compliance frameworks (HIPAA, SOC 2, GDPR, etc.) across the data platform.
• Performance & Reliability Optimization: Drive observability, automation, and performance tuning across data pipelines and infrastructure to ensure reliability at scale.
• Cross-Functional Collaboration: Partner with product, analytics, compliance, and infrastructure teams to align data architecture with business goals and regulatory requirements.
• Innovation & Thought Leadership: Stay ahead of industry trends, evaluate emerging technologies, and contribute to strategic decisions on platform evolution, including generative AI integration and event-driven systems.

Data Science and Machine Learning - Data Science and Machine Learning - Python Database - Database Programming - SQL Data & AI - Data Engineering - Data Quality & Validation Beh - Communication and collaboration

Perks and Benefits for Irisians

Iris provides world-class benefits for a personalized employee experience. These benefits are designed to support financial, health and well-being needs of Irisians for a holistic professional and personal growth. Click here to view the benefits.