Data Engineer Lead

Iris Software, Inc.; Dadri, Pune District, Gurugram District; Hybrid; INR 900,000 - 1,400,000; Full time

Job summary

Iris Software, Inc. is seeking a data engineer to design scalable data processing solutions using PySpark and modern distributed frameworks. You will define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.

Collaborate with cross-functional teams to ensure high-quality data for analytics, governance, and streaming pipelines using AWS services like Lambda, Glue, and Kinesis.

Qualifications

• Must have strong Python and advanced SQL skills.
• Experience with Spark/PySpark and distributed data processing.
• Ability to design data ingestion, transformation, and processing architectures.
• Experience with cloud platforms (AWS) and data pipelines.
• Knowledge of data quality and validation practices.

Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.

Skills

Python

Advanced SQL

PySpark

Spark / distributed processing

Data Quality

Ingestion Architecture

Tools

AWS (Lambda, Glue, EMR, DynamoDB)

S3

Redshift

Kafka / Kinesis

Databricks

Job description

Mandatory Skills:

Python, Apache Spark, Data Quality & Validation, PySpark, SQL, Amazon Kinesis, Apache Kafka, Databricks Workflows, Delta Lake on Databricks

Additional Skills:

AWS S3 + Amazon Aurora + DynamoDB, AWS Glue, AWS Redshift, Data Governance Frameworks, GitHub

Key Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.

Behavioral Competencies

• Demonstrates strong ownership while driving data engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.

Must Have

• Python & Advanced SQL
• Spark / distributed processing / PySpark
• Cloud platforms (AWS) Modules like Lambda, Glue, EMR, Dynamo DB, S3, Redshift, EC2, RDS, Step Functions, Cloud Watch, Systems Manager, Event Bridge, DMS
• SSIS and experience with C# scripting
• Data pipeline automation

Nice to have

• Kafka/Kinesis streaming
• Basic knowledge of AI
• Power BI
• Postgre SQL
• DBT