Databricks - Lead

Iris Software; Dadri; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Iris Software in Noida is seeking a Databricks - Lead to define and drive enterprise data engineering strategy aligned with data modernization initiatives. You will establish standards, governance, and best practices across teams, and lead the design of scalable data processing architectures using PySpark and modern data platform technologies.

The role focuses on Snowflake and Delta Lake, real-time architectures with Kafka or Kinesis, data quality and governance, and collaboration with

Qualifications

• Must be able to define and drive enterprise data engineering strategy.
• Experience establishing governance standards and best practices across teams.
• Strong design leadership for enterprise-scale data platforms using PySpark.

Responsibilities

• Define and drive enterprise data engineering strategy aligned with data modernization.
• Lead design of enterprise-scale data processing architectures using PySpark and modern data platforms.
• Define standards for Snowflake and Delta Lake-based platforms supporting analytics and ops workloads.
• Drive real-time and event-driven data architectures using Kafka or Kinesis.
• Establish governance across ingestion, transformation, streaming, and processing frameworks.

Job description

Job Title

Databricks - Lead

Location

Noida

Responsibilities

• Define and drive enterprise data engineering strategy aligned with organizational objectives and data modernization initiatives.
• Establish data engineering standards, governance frameworks, and best practices across teams.
• Lead the design of enterprise-scale data processing architectures using PySpark and modern data platform technologies.
• Define enterprise standards for Snowflake and Delta Lake-based data platforms supporting analytical and operational workloads.
• Drive real-time and event-driven data architecture initiatives using Apache Kafka or Amazon Kinesis.
• Establish governance standards for data ingestion, transformation, streaming, and processing frameworks.
• Define workflow orchestration, scheduling, and operational governance standards using Apache Airflow or Databricks Workflows.
• Establish data quality, validation, monitoring, and operational excellence frameworks across data engineering ecosystems.
• Define enterprise standards for data products, data quality ownership, metadata management, discoverability, and trusted business data consumption across the organization.
• Establish architecture standards for modern Lakehouse platforms, data observability, platform engineering, and scalable cloud-native data ecosystems supporting enterprise analytics and AI initiatives.
• Define AI-ready data foundation strategies supporting structured and unstructured data processing, vector-enabled architectures, retrieval patterns, and future GenAI and Agentic AI initiatives.
• Partner with business stakeholders to translate business objectives into scalable data platform capabilities, data products, and enterprise data architecture decisions.
• Drive adoption of AI-assisted engineering practices across data engineering teams to improve developer productivity, code quality, documentation, testing, and delivery effectiveness while maintaining governance standards.
• Lead architecture reviews and ensure data solutions meet scalability, reliability, maintainability, and performance objectives.
• Guide teams on distributed data processing, streaming architectures, modern data platforms, and engineering best practices.
• Identify platform risks, scalability bottlenecks, operational gaps, and architectural challenges while defining mitigation strategies.
• Collaborate with various teams and leadership stakeholders to align data initiatives with organizational objectives.
• Drive continuous improvement initiatives focused on platform maturity, engineering excellence, scalability, reliability, and delivery effectiveness.

Behavioral Competencies

• Demonstrates leadership and accountability in driving data engineering excellence across programs and initiatives.
• Collaborate effectively with various teams and business stakeholders to ensure smooth delivery.
• Promotes a culture of innovation, quality, continuous improvement, and data engineering excellence.
• Apply strategic thinking to address platform challenges, scalability risks, and business priorities.
• Demonstrates strong decision-making while balancing performance, scalability, reliability, and delivery objectives.
• Communicates effectively regarding platform strategy, risks, dependencies, and operational outcomes.

Mandatory Competencies

• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Data AI - Data Engineering - Data Quality Validation
• Big Data - Big Data - Pyspark
• Database - Database Programming - SQL
• Data AI - Data Engineering - Apache Kafka
• Data Science and Machine Learning - Data Science and Machine Learning - Databricks
• Beh - Communication and collaboration