Data Eng Assoc Prin Engineer

PepsiCo; Hyderabad; Hybrid; INR 2,500,000 - 5,000,000; Full time

Job summary

PepsiCo seeks a data engineering lead to drive the design and operations of its data products across revenue management, supply chain, manufacturing, and logistics. You will build and scale data pipelines from diverse sources into the PepsiCo Data Lake, enabling analytics, visualization, machine learning, and product development for business impact.

You will lead a team of data engineers in a hybrid environment, collaborating with process owners, product teams and business users to implement

Qualifications

• 9+ years in technology; 5+ years in software development, data engineering, and systems architecture.
• 4+ years in Data Lake, Data Warehousing, and Data Analytics tools.
• 4+ years SQL optimization and programming in Python, PySpark, and Scala.
• 3+ years cloud data engineering in Azure.
• Proficiency with Azure cloud services; Azure Certification is a plus.
• Experience with data modeling and building high-volume ETL/ELT pipelines.
• Experience with cloud-scale apps and Kubernetes.

Responsibilities

• Manage and scale data pipelines from internal and external sources to support new product launches and drive data quality across data products.
• Create reusable accelerators and solutions to migrate data from legacy data warehouse platforms such as Teradata to Azure Databricks and Azure SQL.
• Enable and accelerate standards-based development prioritizing reuse of code, adopt test-driven development, unit testing and test automation with end-to-end observability of data
• Build and own the automation and monitoring frameworks that captures metrics and operational KPIs for data pipeline quality, performance and cost.
• Collaborate with internal clients (product teams, sector leads, data science teams) and external partners (SI partners/data providers) to drive solutioning and clarify solution requirements.
• Define and manage SLA’s for data products and processes running in production.

Skills

Data pipelines

Python

PySpark

Scala

SQL optimization

Test-driven development

Observability

Kubernetes

Azure Databricks

Data modeling

Tools

Azure Databricks

Kubernetes

Azure SQL

Job description

Overview

As a data engineering lead, you will be the key technical expert overseeing PepsiCo's data product build & operations and drive a strong vision for how data engineering can proactively create a positive impact on the business. You'll be empowered to create & lead a strong team of data engineers who build data pipelines into various source systems, rest data on the PepsiCo Data Lake, and enable exploration and access for analytics, visualization, machine learning, and product development efforts across the company. As a member of the data engineering team, you will help lead the development of very large and complex data applications into public cloud environments directly impacting the design, architecture, and implementation of PepsiCo's flagship data products around topics like revenue management, supply chain, manufacturing, and logistics. You will work closely with process owners, product owners and business users. You'll be working in a hybrid environment with in-house, on-premises data sources as well as cloud and remote systems.

Responsibilities

• Manage and scale data pipelines from internal and external data sources to support new product launches and drive data quality across data products.
• Create reusable accelerators and solutions to migrate data from legacy data warehouse platforms such as Teradata to Azure Databricks and Azure SQL.
• Enable and accelerate standards-based development prioritizing reuse of code, adopt test-driven development, unit testing and test automation with end-to-end observability of data
• Build and own the automation and monitoring frameworks that captures metrics and operational KPIs for data pipeline quality, performance and cost.
• Collaborate with internal clients (product teams, sector leads, data science teams) and external partners (SI partners/data providers) to drive solutioning and clarify solution requirements.
• Define and manage SLA’s for data products and processes running in production.

Qualifications

• 9+ years of overall technology experience that includes at least 5+ years of hands-on software development, data engineering, and systems architecture.
• 4+ years of experience with Data Lake Infrastructure, Data Warehousing, and Data Analytics tools.
• 4+ years of experience in SQL optimization and performance tuning, and development experience in programming languages like Python, PySpark, Scala etc.).
• 3+ years in cloud data engineering experience in Azure.
• Fluent with Azure cloud services. Azure Certification is a plus.
• Experience with data modelling, data warehousing, and building high-volume ETL/ELT pipelines.
• Experience with running and scaling applications on the cloud infrastructure and containerized services like Kubernetes.