Data Engineer Python - Senior Engineer

Iris Software; Dadri; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Iris Software in Noida, India is seeking a Senior Data Engineer to design and implement scalable data pipelines using PySpark, Kafka, and Databricks. This role focuses on building reliable lakehouse architectures and processing large-scale data for analytics platforms.

The position requires guiding end-to-end data ingestion, transformation, and governance, while mentoring teammates on modern data engineering practices and tools. Notably, experience with Delta Lake on Databricks is essential.

Qualifications

• Proficient in PySpark for scalable data processing.
• Experience with Apache Kafka for streaming data pipelines.
• Familiarity with Databricks Workflows for orchestrating jobs.
• Knowledge of Delta Lake on Databricks for reliable lakehouse storage.

Responsibilities

• Design scalable data engineering solutions using PySpark and distributed frameworks.
• Define ingestion, transformation, and processing architectures to meet analytics goals.
• Design and optimize Delta Lake on Databricks solutions for enterprise-scale platforms.
• Lead development of high-performance batch and streaming data pipelines.
• Architect event-driven data architectures using Kafka or Kinesis.
• Establish monitoring, scheduling, and controls for reliable pipelines.

Tools

PySpark

Apache Kafka

Databricks Workflows

Delta Lake on Databricks

Job description

Data Engineer Python - Senior Engineer

Location: Noida, UP, India

Job Description

Mandatory Skills

• PySpark
• Apache Kafka
• Databricks Workflows
• Delta Lake on Databricks

Responsibilities

• Design scalable data engineering solutions using PySpark and modern distributed data processing frameworks.
• Define data ingestion, transformation, and processing architectures aligned with business and analytical objectives.
• Design and optimize Snowflake or Delta Lake on Databricks solutions to support enterprise-scale data platforms.
• Lead implementation of high-performance batch and streaming data pipelines.
• Design and optimize event-driven data architectures using Apache Kafka or Amazon Kinesis.
• Define data streaming standards, integration frameworks, and scalable processing patterns.
• Architect workflow orchestration solutions using Apache Airflow or Databricks Workflows.
• Establish monitoring, scheduling, and operational controls for reliable pipeline execution.
• Drive data quality, validation, reconciliation, and governance practices across data engineering solutions.
• Design data engineering solutions following modern Lakehouse architecture principles, data observability practices, and platform engineering standards to improve scalability, reliability, and operational visibility.
• Drive development of business-focused data products by improving data quality, discoverability, usability, documentation, and trusted data consumption across analytical platforms.
• Promote responsible use of AI-assisted engineering capabilities to improve development productivity, testing, documentation, and engineering quality.
• Review data pipeline designs and implementations to ensure adherence to engineering, scalability, and performance standards.
• Troubleshoot complex data processing, workflow, and streaming platform issues through detailed root cause analysis.
• Mentor team members on PySpark, Snowflake, Delta Lake, Kafka, Kinesis, Airflow, and data engineering best practices.
• Collaborate with various teams and stakeholders to support end-to-end data platform delivery.

Behavioral Competencies

• Demonstrates strong ownership while driving data engineering excellence.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes quality-focused engineering through proactive validation, optimization, and continuous improvement.
• Apply strong analytical thinking to evaluate complex data engineering and platform challenges.
• Demonstrate adaptability while managing evolving technologies, data ecosystems, and business requirements.
• Communicates effectively regarding delivery status, risks, dependencies, and improvement opportunities.
• Maintains high attention to detail across data architecture, pipeline design, testing, and implementation activities.
• Encourages continuous improvement in data engineering practices and platform operations.
• Supports knowledge sharing and mentoring to strengthen team capabilities.
• Balances scalability, performance, reliability, and business priorities while driving delivery excellence.
• Promotes innovation by adopting modern data engineering practices, platform engineering principles, and AI-assisted development approaches to improve engineering productivity and solution quality.

Mandatory Competencies

• Data Science and Machine Learning - Data Science and Machine Learning - Python
• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Database - Database Programming - SQL
• Data AI - Data Engineering - Data Quality Validation
• Big Data - Big Data - Pyspark
• Data AI - Data Engineering - Apache Kafka
• Data Science and Machine Learning - Data Science and Machine Learning - Databricks
• Beh - Communication and collaboration