Principal Data Engineer

iCIMS, Inc.; Hyderabad; On-site; INR 3,500,000 - 6,500,000; Full time

Job summary

iCIMS, Inc. is seeking a highly experienced Data Engineering leader to build the next generation of our Talent Cloud platform with scalable data pipelines, modern Lakehouse architectures, and analytics infrastructure powering data-driven decisions and AI capabilities.

You will lead design, development, and optimization of enterprise data solutions, partnering with software engineers, data scientists, product leaders, and business stakeholders in a culture that values ownership and continuous

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or related field (or equivalent professional experience).
• 7+ years of experience building enterprise-scale data pipelines, platforms, and distributed data systems.
• Expert-level experience with Databricks and modern Lakehouse architectures.
• Deep expertise in Apache Spark, preferably PySpark, for large-scale distributed data processing.
• Strong hands‑on experience with Delta Lake, Unity Catalog, Databricks Workflows, and enterprise Databricks deployments.
• Proficiency in Python and strong familiarity with Java.
• Advanced SQL skills and experience with relational and non‑relational databases such as SQL Server, PostgreSQL, MySQL, and MongoDB.
• Strong expertise with cloud platforms, preferably AWS, including services such as S3, Redshift, IAM, and cloud‑native data architectures.
• Experience designing and implementing modern data lakes, data warehouses, and Lakehouse solutions.
• Expertise with streaming platforms such as Kafka and AWS Kinesis and event‑driven architectures.
• Strong understanding of data modeling, warehousing, schema design, and data lifecycle management.
• Experience with data transformation tools such as dbt and business intelligence platforms such as Looker.
• Experience building and exposing APIs for data consumption and integration.
• Strong understanding of Git, CI/CD pipelines, infrastructure automation, and software engineering best practices.
• Experience implementing enterprise data governance, security, and compliance controls across modern data platforms.
• Experience optimizing Databricks clusters, Spark workloads, storage formats, and platform costs.
• Strong analytical, troubleshooting, and problem-solving skills with a passion for continuous learning and innovation.
• Excellent communication and collaboration skills across technical and non-technical audiences.
• Demonstrated experience mentoring engineers, leading technical projects, and influencing architectural decisions.

Responsibilities

• Lead a team of data engineers, providing technical direction, mentorship, and performance management.
• Design, develop, and maintain scalable data pipelines to collect, process, and store data from multiple sources.
• Architect, build, and optimize Databricks Lakehouse solutions supporting enterprise analytics, reporting, AI/ML, and data products.
• Develop large-scale distributed data processing solutions using Databricks, Apache Spark (PySpark), and Delta Lake.
• Implement and manage Medallion Architecture (Bronze, Silver, Gold) data layers to improve data quality, governance, and usability.
• Build and optimize data infrastructure to support analytics, reporting, and AI/ML workloads at scale.
• Implement event sourcing and streaming architectures using platforms such as Kafka and AWS Kinesis for real-time data processing.
• Leverage Databricks capabilities such as Auto Loader, Delta Live Tables, Databricks Workflows, Unity Catalog, and MLflow to improve platform efficiency and governance.
• Establish best practices for Databricks workspace administration, cluster management, performance tuning, monitoring, and cost optimization.
• Apply data governance, security principles, and compliance frameworks to ensure data quality and regulatory adherence.
• Collaborate with data scientists, software engineers, analytics teams, and product leaders to deliver scalable and reliable data solutions.
• Drive the adoption of modern data engineering practices, frameworks, and tooling across the organization.
• Troubleshoot and resolve complex data-related issues while maintaining data quality, availability, and integrity.
• Contribute to architectural decisions and technology roadmaps for the organization's data platform.
• Mentor data engineers and provide technical leadership across multiple initiatives and teams.

Skills

Mentoring engineers

Leadership

Communication

Problem solving

Analytical thinking

Education

Bachelor's degree in Computer Science, Engineering, Data Science, or related field

Equivalent professional experience

Tools

Databricks

Delta Lake

Unity Catalog

Databricks Workflows

MLflow

Kafka

AWS

S3

Redshift

dbt

Looker

Git

CI/CD

Job description

Job Overview

At iCIMS, we're redefining how people connect with opportunity through intelligent, human-centered technology. We're growing rapidly and are seeking a highly experienced Data Engineering leader to build the next generation of our Talent Cloud platform through scalable data pipelines, modern data architectures, and analytics infrastructure that power data-driven decision-making and AI capabilities.

This role requires deep expertise in Databricks and modern Lakehouse architectures, leveraging Apache Spark, Delta Lake, and cloud-native technologies to build scalable and reliable data platforms. You will lead the design, development, and optimization of enterprise data solutions while partnering with software engineers, data scientists, product leaders, and business stakeholders. This position combines hands‑on technical leadership, architectural ownership, and team mentorship within a culture that values innovation, ownership, and continuous learning.

Responsibilities

• Lead a team of data engineers, providing technical direction, mentorship, and performance management.
• Design, develop, and maintain scalable data pipelines to collect, process, and store data from multiple sources.
• Architect, build, and optimize Databricks Lakehouse solutions supporting enterprise analytics, reporting, AI/ML, and data products.
• Develop large-scale distributed data processing solutions using Databricks, Apache Spark (PySpark), and Delta Lake.
• Implement and manage Medallion Architecture (Bronze, Silver, Gold) data layers to improve data quality, governance, and usability.
• Build and optimize data infrastructure to support analytics, reporting, and AI/ML workloads at scale.
• Implement event sourcing and streaming architectures using platforms such as Kafka and AWS Kinesis for real-time data processing.
• Leverage Databricks capabilities such as Auto Loader, Delta Live Tables, Databricks Workflows, Unity Catalog, and MLflow to improve platform efficiency and governance.
• Establish best practices for Databricks workspace administration, cluster management, performance tuning, monitoring, and cost optimization.
• Apply data governance, security principles, and compliance frameworks to ensure data quality and regulatory adherence.
• Collaborate with data scientists, software engineers, analytics teams, and product leaders to deliver scalable and reliable data solutions.
• Drive the adoption of modern data engineering practices, frameworks, and tooling across the organization.
• Troubleshoot and resolve complex data-related issues while maintaining data quality, availability, and integrity.
• Contribute to architectural decisions and technology roadmaps for the organization's data platform.
• Mentor data engineers and provide technical leadership across multiple initiatives and teams.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, or related field (or equivalent professional experience).
• 7+ years of experience building enterprise-scale data pipelines, platforms, and distributed data systems.
• Expert-level experience with Databricks and modern Lakehouse architectures.
• Deep expertise in Apache Spark, preferably PySpark, for large-scale distributed data processing.
• Strong hands‑on experience with Delta Lake, Unity Catalog, Databricks Workflows, and enterprise Databricks deployments.
• Proficiency in Python and strong familiarity with Java.
• Advanced SQL skills and experience with relational and non‑relational databases such as SQL Server, PostgreSQL, MySQL, and MongoDB.
• Strong expertise with cloud platforms, preferably AWS, including services such as S3, Redshift, IAM, and cloud‑native data architectures.
• Experience designing and implementing modern data lakes, data warehouses, and Lakehouse solutions.
• Expertise with streaming platforms such as Kafka and AWS Kinesis and event‑driven architectures.
• Strong understanding of data modeling, warehousing, schema design, and data lifecycle management.
• Experience with data transformation tools such as dbt and business intelligence platforms such as Looker.
• Experience building and exposing APIs for data consumption and integration.
• Strong understanding of Git, CI/CD pipelines, infrastructure automation, and software engineering best practices.
• Experience implementing enterprise data governance, security, and compliance controls across modern data platforms.
• Experience optimizing Databricks clusters, Spark workloads, storage formats, and platform costs.
• Strong analytical, troubleshooting, and problem-solving skills with a passion for continuous learning and innovation.
• Excellent communication and collaboration skills across technical and non-technical audiences.
• Demonstrated experience mentoring engineers, leading technical projects, and influencing architectural decisions.