Principal Data Engineer

Nielseniq India; Pune District; On-site; INR 4,000,000 - 7,000,000; Full time

Benefits offered by this job

Flexible working environment

Volunteer time off

LinkedIn Learning

Employee-Assistance-Program (EAP)

Job summary

NielsenIQ India seeks a Principal Software Engineer to provide technical leadership across data engineering, data science and software teams. You will bridge data platforms, mentor engineers, and drive best practices in cloud-native design. The role involves line management and hands-on implementation in Azure and Databricks.

Lead architecture for scalable data solutions, champion CI/CD, governance and secure, maintainable pipelines, with strong collaboration across Product and DevOps teams.

Qualifications

• 10+ years of strong software development experience.
• Proficient in Python, Scala, Java or similar languages.
• Experience with Azure-based data platforms (Databricks, Azure Data Lake, Azure Synapse, Azure Functions).
• Expertise in Apache Spark and Delta Lake; large-scale data pipelines.
• Experience with Apache Airflow.
• Cloud experience in Azure; containerization (Docker, Kubernetes) and IaC (Terraform, Bicep).
• Knowledge of microservices, event-driven systems, distributed computing.
• Experience leading teams in an agile environment with CI/CD, automated testing, DevOps.
• Excellent problem-solving and communication; engage with stakeholders.
• Experience in data engineering, data science, or analytics-heavy environments desirable.
• Proven line management experience.

Responsibilities

• Act as a technical architect aligning multiple teams on data engineering and data science.
• Design scalable data processing solutions on Azure and Databricks.
• Drive CI/CD, testing strategies, and cloud-native architecture.
• Lead hands-on development, code reviews, and POC work.
• Mentor developers and manage line of sight and performance.
• Collaborate with Product Owners and Engineering Managers to translate business needs.
• Guide DevOps to improve deployment pipelines and observability.
• Ensure governance and security across the data platform.
• Engage stakeholders with technical insights.

Skills

Technical leadership

Azure Databricks

Python/Scala/Java

CI/CD

Data pipelines

Data governance

Cloud security

Agile

Team mentoring

DevOps

Tools

Docker

Kubernetes

Terraform

Bicep

Airflow

Delta Lake

Apache Spark

Databricks

Azure Data Lake

Azure Synapse

Job description

We are looking for aPrincipal Software Engineerto provide technical leadership within our development teams, ensuring best practices, architectural coherence, and effective collaboration across projects. This role is ideal for a highly experienced engineer who can bridge the gap between data engineering, data science, and software engineering, helping teams build scalable, maintainable, and well-structured data solutions.

As aPrincipal Software Engineer, you will play a hands-on role in designing and implementing solutions while mentoring developers, influencing technical direction, and driving best practices in software and data engineering. This role includesline management responsibilities, ensuring the growth and development of team members. The role will be working within anAzure and Databricks environment, leveraging the power of cloud-native technologies and modern data platforms.

Key Responsibilities

Technical Leadership & Architecture

• Act as a technical architect, ensuring alignment between the work of multiple development teams in data engineering and data science.
• Design scalable, high-performance data processing solutions withinAzure and Databricks, considering factors such as governance, security, and maintainability.
• Drive the adoption of best practices in software development, including CI/CD, testing strategies, and cloud-native architecture.
• Work closely with Product Owners and Engineering Managers to translate business needs into technical solutions.

Hands-on Development & Technical Excellence

• Lead by example through high-quality coding, code reviews, and proof-of-concept development.
• Solve complex engineering problems and contribute to critical design decisions.
• Ensure effective use ofAzure services,Databricks, and data processing frameworks such asApache Spark, Delta Lake, and Airflow.
• Develop and optimisedata pipelines, data transformations, and ML workflowsin a cloud environment.

Line Management & Team Development

• Provide line management to engineers, ensuring their professional growth and development.
• Conduct performance reviews, set development goals, and mentor team members to enhance their skills.
• Foster a collaborative and high-performing engineering culture, promoting knowledge sharing and continuous improvement.
• Support hiring, onboarding, and career development initiatives within the engineering team.

Collaboration & Cross-Team Coordination

• Act as the technical glue betweendata engineers, data scientists, and software developers, ensuring smooth integration of different components.
• Provide mentorship and guidance to developers, helping them level up their skills and technical understanding.
• Work with DevOps teams to improve deployment pipelines, observability, and infrastructure as code.
• Engage with stakeholders across the business, translating technical concepts into business-relevant insights.

Governance, Security & Data Best Practices

• Championdata governance, lineage, and securityacross the platform.
• Advocate for and implement scalabledata architecture patterns, such asData Mesh, Lakehouse, or event-driven pipelines.
• Ensure compliance with industry standards, internal policies, and regulatory requirements.

Requirements & Experience

• 10+ years of strong software development experience required
• Strong software engineering background with experience in designing and building production-grade applications inPython, Scala, Java, or similar languages.
• Proven experience withAzure-based data platforms, specificallyDatabricks, Azure Data Lake, Azure Synapse, and Azure Functions.
• Expertise inApache SparkandDelta Lake, with experience building large-scale distributed data pipelines.
• Experience withworkflow orchestration toolslikeApache Airflow.
• Cloud experience inAzure, includingcontainerisation (Docker, Kubernetes)andinfrastructure as code (Terraform, Bicep).
• Strong knowledge of modern software architecture, includingmicroservices, event-driven systems, and distributed computing.
• Experience leading teams in anagileenvironment, with a strong understanding ofCI/CD pipelines, automated testing, and DevOps practices.
• Excellent problem-solving and communication skills, with the ability to engage with both technical and non-technical stakeholders.
• Previous experience indata engineering, data science, or analytics-heavy environmentsis highly desirable.
• Provenline management experience, including mentoring, career development, and performance management of engineering teams.

Additional Information

Our Benefits

• Flexible working environment
• Volunteer time off
• LinkedIn Learning
• Employee-Assistance-Program (EAP)