ETL Data - Senior Engineer

Iris Software, Inc.; Dadri; On-site; INR 3,000,000 - 4,200,000; Full time

Job summary

Iris Software in India is seeking a data engineering specialist to design and implement scalable data pipelines using PySpark and Databricks. You will lead ingestion, transformation, and governance across analytics platforms to support enterprise-scale data needs.

The role emphasizes designing Lakehouse solutions, ensuring data quality, and enabling trusted data consumption for business insights. On-site opportunity in Uttar Pradesh.

Qualifications

• Mandatory skills include PySpark, Databricks Workflows, Delta Lake on Databricks and Amazon Kinesis.

Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions to support enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.
• Define data streaming standards, integration frameworks, and scalable processing patterns.
• Architect workflow orchestration solutions using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices across data engineering solutions.
• Design data engineering solutions following modern Lakehouse architecture principles, data observability practices, and platform engineering standards to improve scalability, reliability, and operational visibility.
• Drive development of business-focused data products by improving data quality, discoverability, usability, documentation, and trusted data consumption across analytical platforms.

Skills

PySpark

Databricks Workflows

Delta Lake on Databricks

Amazon Kinesis

Tools

Snowflake

Databricks

Job description

Why Join Iris?
Are you ready to do the best work of your career at one ofIndia’s Top 25 Best Workplaces in IT industry? Do you want to grow in an award-winning culture thattruly values your talent and ambitions?
Join Iris Software — one offastest-growing IT services companies— whereyou own and shape your success story.

About Us
At Iris Software, our vision is to be our client’s most trusted technology partner, and the first choice for the industry’s top professionals to realize their full potential.

With over 4,300 associates across India, U.S.A, and Canada, we help our enterprise clients thrive with technology-enabled transformation across financial services, healthcare, transportation & logistics, and professional services.

Our work covers complex, mission-critical applications with the latest technologies, such as high-value complex Application & Product Engineering, Data & Analytics, Cloud, DevOps, Data & MLOps, Quality Engineering, and Business Automation.

Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark

Data & AI - Data Engineering - Data Quality & Validation

Data Science and Machine Learning - Data Science and Machine Learning - Python

Database - Database Programming - SQL

Cloud - AWS - AWS SNS, AWS SQS, AWS Kinesis

Data Science and Machine Learning - Data Science and Machine Learning - Databricks

Cloud - AWS - Amazon CloudWatch

Cloud - AWS - Amazon IAM, AWS Secrets Manager, AWS KMS, AWS Cognito

Cloud - AWS - Amazon API Gateway

Cloud - AWS - Tensorflow on AWS, AWS Glue, AWS EMR, Amazon Data Pipeline, AWS Redshift

Cloud - AWS - Amazon EC2 / Autoscalaing / Load Balancing, AWS App Runner

Beh - Communication and collaboration

Job Description

Mandatory Skills: PySpark,Databricks Workflows,Delta Lake on Databricks,Amazon Kinesis

Key Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions to support enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.
• Define data streaming standards, integration frameworks, and scalable processing patterns.
• Architect workflow orchestration solutions using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices across data engineering solutions.
• Design data engineering solutions following modern Lakehouse architecture principles, data observability practices, and platform engineering standards to improve scalability, reliability, and operational visibility.
• Drive development of business-focused data products by improving data quality, discoverability, usability, documentation, and trusted data consumption across analytical platforms.
• Promote responsible use of AI-assisted engineering capabilities to improve development productivity, testing, documentation, and engineering quality.
• Review data pipeline designs and implementations to ensure adherence to engineering, scalability, and performance standards.
• Troubleshoot complex data processing, workflow, and streaming platform issues through detailed root cause analysis.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.

Behavioral Competencies

• Demonstrates strong ownership while driving data engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes quality-focused engineering through proactive validation, optimization, and continuous improvement.
• Apply strong analytical thinking to evaluate complex data engineering and platform challenges.
• Demonstrate adaptability while managing evolving technologies, data ecosystems, and business requirements.
• Communicates effectively regarding delivery status, risks, dependencies, and improvement opportunities.
• Maintains high attention to detail across data architecture, pipeline design, testing, and implementation activities.
• Encourages continuous improvement in data engineering practices and platform operations.
• Supports knowledge sharing and mentoring to strengthen team capabilities.
• Balances scalability, performance, reliability, and business priorities while driving delivery excellence.
• Promotes innovation by adopting modern data engineering practices, platform engineering principles, and AI-assisted development approaches to improve engineering productivity and solution quality.

Perks and Benefits for Irisians
Iris provides world-class benefits for a personalized employee experience. These benefits are designed to support financial, health and well-being needs of Irisians for a holistic professional and personal growth.