Data Engineer - Consultant

Iris Software; Dadri; On-site; INR 1,400,000 - 2,200,000; Full time

Job summary

Iris Software in Noida is seeking a Data Engineer - Consultant to design and implement scalable data pipelines and modern Lakehouse architectures. You will work with PySpark, Apache Airflow, Kafka, Delta Lake on Databricks, and Snowflake to build reliable ingestion, transformation, and processing layers aligned with business needs.

You will mentor juniors, implement batch and streaming pipelines, ensure data quality and governance, and collaborate with cross-functional teams to deliver

Qualifications

• Experience designing scalable data engineering solutions using PySpark and distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures to meet business and analytical goals.
• Experience with Snowflake or Delta Lake on Databricks for enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.

Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions to support enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.
• Design and optimize event-driven data architectures using Apache Kafka or Amazon Kinesis.
• Define data streaming standards, integration frameworks, and scalable processing patterns.
• Architect workflow orchestration solutions using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices across data engineering solutions.
• Design data engineering solutions following modern Lakehouse architecture principles, data observability practices, and platform engineering standards to improve scalability, reliability, and operational visibility.
• Drive development of business-focused data products by improving data quality, discoverability, usability, documentation, and trusted data consumption across analytical platforms.
• Promote responsible use of AI-assisted engineering capabilities to improve development productivity, testing, documentation, and engineering quality.
• Review data pipeline designs and implementations to ensure adherence to engineering, scalability, and performance standards.
• Troubleshoot complex data processing, workflow, and streaming platform issues through detailed root cause analysis.
• Mentor team members on PySpark, Snowflake, Delta Lake, Kafka, Kinesis, Airflow, and data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.

Skills

Apache Airflow

PySpark

Apache Kafka

Delta Lake on Databricks

Airflow

Snowflake

Databricks

Kinesis

Tools

Snowflake

Databricks

Airflow

Kinesis

Job description

Data Engineer - Consultant

Location: Noida

Company: Iris Software

Mandatory Skills

• Apache Airflow,PySpark,Apache Kafka,Delta Lake on Databricks

Key Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions to support enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.
• Design and optimize event-driven data architectures using Apache Kafka or Amazon Kinesis.
• Define data streaming standards, integration frameworks, and scalable processing patterns.
• Architect workflow orchestration solutions using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices across data engineering solutions.
• Design data engineering solutions following modern Lakehouse architecture principles, data observability practices, and platform engineering standards to improve scalability, reliability, and operational visibility.
• Drive development of business-focused data products by improving data quality, discoverability, usability, documentation, and trusted data consumption across analytical platforms.
• Promote responsible use of AI-assisted engineering capabilities to improve development productivity, testing, documentation, and engineering quality.
• Review data pipeline designs and implementations to ensure adherence to engineering, scalability, and performance standards.
• Troubleshoot complex data processing, workflow, and streaming platform issues through detailed root cause analysis.
• Mentor team members on PySpark, Snowflake, Delta Lake, Kafka, Kinesis, Airflow, and data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.

Behavioral Competencies

• Demonstrates strong ownership while driving data engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes quality-focused engineering through proactive validation, optimization, and continuous improvement.
• Apply strong analytical thinking to evaluate complex data engineering and platform challenges.
• Demonstrate adaptability while managing evolving technologies, data ecosystems, and business requirements.
• Communicates effectively regarding delivery status, risks, dependencies, and improvement opportunities.
• Maintains high attention to detail across data architecture, pipeline design, testing, and implementation activities.
• Encourages continuous improvement in data engineering practices and platform operations.
• Supports knowledge sharing and mentoring to strengthen team capabilities.
• Balances scalability, performance, reliability, and business priorities while driving delivery excellence.
• Promotes innovation by adopting modern data engineering practices, platform engineering principles, and AI-assisted development approaches to improve engineering productivity and solution quality.

Mandatory Competencies

• Data Science and Machine Learning - Data Science and Machine Learning - Python
• Database - Database Programming - SQL
• Data & AI - Data Engineering - Data Quality & Validation
• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Big Data - Big Data - Pyspark
• Data & AI - Data Engineering - Apache Kafka
• Programming Language - Python - Apache Airflow
• Cloud - GCP - Cloud Spanner, Cloud SQL, Cloud Storage
• Cloud - GCP - Apigee API Management, API Gateway
• Development Tools and Management - Development Tools and Management - CI/CD
• Cloud - GCP - Cloud SDK, Cloud Shell, Cloud Deployment Manager
• Cloud - GCP - Cloud Build, Google Cloud Deploy, Dataflow, Cloud Run
• Beh - Communication and collaboration