Aws Data Engineer Lead

Iris Software; Dadri, Pune District, Gurugram District; Hybrid; INR 4,000,000 - 7,000,000; Full time

Job summary

Iris Software, Inc. is seeking an AWS Data Engineer with 8-10 years of experience to design and implement enterprise-scale data platforms in Noida.

The role focuses on real-time analytics, data governance, and scalable cloud-native architectures using PySpark, Kinesis, and Delta Lake on Databricks. You will drive standards for data ingestion, processing, and governance while partnering with stakeholders to translate business needs into robust data solutions that enable AI initiatives and

Qualifications

• 8-10 years of data engineering experience with large-scale platforms.
• Experience delivering enterprise-scale data pipelines and lakehouse architectures.
• Strong skills in Spark-based processing, SQL and data governance.

Responsibilities

• Define and drive enterprise data engineering strategy aligned with data modernization goals.
• Establish standards, governance, and best practices across teams.
• Lead design of enterprise-scale data processing architectures using PySpark and modern platforms.
• Define standards for Snowflake and Delta Lake-based data platforms supporting analytics and operations.
• Drive real-time and event-driven data architecture using Apache Kafka or Kinesis.
• Establish governance for ingestion, transformation, streaming, and processing pipelines.
• Define workflow orchestration, scheduling, and governance using Airflow or Databricks Workflows.
• Establish data quality, monitoring, and operational excellence frameworks across data ecosystems.
• Define data products, metadata management, and trusted business data consumption standards.
• Set architecture standards for Lakehouse platforms and scalable cloud-native data ecosystems.
• Define AI-ready data foundations for structured/unstructured data, vectors, retrieval patterns, GenAI/Agentic AI initiatives.
• Collaborate with business stakeholders to translate objectives into scalable data platform capabilities.
• Drive AI-assisted engineering practices to improve productivity, code quality, and governance.
• Lead architecture reviews ensuring scalability, reliability, and performance.
• Guide teams on distributed processing and modern data platforms.
• Identify risks and bottlenecks and define mitigation strategies.
• Coordinate with leadership to align data initiatives with organizational goals.
• Foster continuous improvement in platform maturity and engineering excellence.

Skills

Data engineering leadership

Architecting data platforms

Real-time data processing

Tools

Apache Spark

PySpark

SQL

Databricks Workflows

Delta Lake on Databricks

Amazon Kinesis

Airflow

Snowflake

AWS Glue

AWS S3

AWS EBS

Amazon RDS

DynamoDB

Job description

At Iris Software, our vision is to be our clients most trusted technology partner, and the first choice for the industrys top professionals to realize their full potential.

With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transportation & logistics, and professional services.

Our work covers complex, mission-critical applications with the latest technologies, such as high-value complex Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation.

Working at Iris: Be valued, be inspired, be your best.

Experience:

8-10 Years

Role

AWS Data Engineer

Location:

Noida

Mandatory Skills:

Apache Spark, PySpark, SQL, Databricks Workflows, Delta Lake on Databricks, Amazon Kinesis

Additional Skills:

AWS Glue, AWS S3 + AWS EBS + Amazon RDS + DynamoDB

Key Responsibilities

• Define and drive enterprise data engineering strategy aligned with organizational objectives and data modernization initiatives.
• Establish data engineering standards, governance frameworks, and best practices across teams.
• Lead the design of enterprise-scale data processing architectures using PySpark and modern data platform technologies.
• Define enterprise standards for Snowflake and Delta Lake-based data platforms supporting analytical and operational workloads.
• Drive real-time and event-driven data architecture initiatives using Apache Kafka or Amazon Kinesis.
• Establish governance standards for data ingestion, transformation, streaming, and processing frameworks.
• Define workflow orchestration, scheduling, and operational governance standards using Apache Airflow or Databricks Workflows.
• Establish data quality, validation, monitoring, and operational excellence frameworks across data engineering ecosystems.
• Define enterprise standards for data products, data quality ownership, metadata management, discoverability, and trusted business data consumption across the organization.
• Establish architecture standards for modern Lakehouse platforms, data observability, platform engineering, and scalable cloud-native data ecosystems supporting enterprise analytics and AI initiatives.
• Define AI-ready data foundation strategies supporting structured and unstructured data processing, vector-enabled architectures, retrieval patterns, and future GenAI and Agentic AI initiatives.
• Partner with business stakeholders to translate business objectives into scalable data platform capabilities, data products, and enterprise data architecture decisions.
• Drive adoption of AI-assisted engineering practices across data engineering teams to improve developer productivity, code quality, documentation, testing, and delivery effectiveness while maintaining governance standards.
• Lead architecture reviews and ensure data solutions meet scalability, reliability, maintainability, and performance objectives.
• Guide teams on distributed data processing, streaming architectures, modern data platforms, and engineering best practices.
• Identify platform risks, scalability bottlenecks, operational gaps, and architectural challenges while defining mitigation strategies.
• Collaborate with various teams and leadership stakeholders to align data initiatives with organizational objectives.
• Drive continuous improvement initiatives focused on platform maturity, engineering excellence, scalability, reliability, and delivery effectiveness.

Behavioral Competencies

• Demonstrates leadership and accountability in driving data engineering excellence across programs and initiatives.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes a culture of innovation, quality, continuous improvement, and data engineering excellence.
• Apply strategic thinking to address platform challenges, scalability risks, and business priorities.
• Demonstrates strong decision-making while balancing performance, scalability, reliability, and delivery objectives.
• Communicates effectively regarding platform strategy, risks, dependencies, and operational outcomes.
• Maintains a proactive approach toward engineering governance, platform standards, and solution excellence.