Senior Data Engineer

Bayer; Bengaluru; On-site; INR 2,200,000 - 3,400,000; Full time

Job summary

Bayer in Bengaluru, India is seeking a senior Data Engineer to design and maintain scalable data pipelines using Python, PySpark, and Databricks. You will implement DataOps, governance, and automated testing while optimizing compute costs with IaC (Terraform) and Azure DevOps.

Collaboration across Finance, Supply Chain, and other functions is essential. Five+ years of data engineering experience with strong Databricks expertise are required, including Delta Lake and Unity Catalog familiarity.

Qualifications

• Five+ years in data engineering, DataOps, DevOps, or a comparable technical role.
• Advanced hands-on experience with Python, PySpark, SQL, Git, and Databricks.
• Strong Databricks ecosystem knowledge: Delta Lake, Unity Catalog, and Serverless SQL.
• Experience with IaC and CI/CD tooling across major platforms.
• Experience implementing data-quality controls, contracts, and automated testing.
• Knowledge of Azure cloud services; AWS/GCP beneficial.

Responsibilities

• Design, build, and maintain scalable, reliable, cost-efficient data pipelines.
• Develop idempotent ETL/ELT pipelines using Python, PySpark, Delta Lake, and MERGE INTO patterns.
• Drive automation across deployments, monitoring, incident remediation, and operations.
• Implement DataOps practices with data contracts, data-quality checks, and tests.
• Establish data governance, lineage tracking, and security access controls with Unity Catalog.
• Define and manage IaC using Terraform and Azure DevOps for Databricks workspaces and storage.
• Build and maintain CI/CD pipelines using Databricks Asset Bundles, Databricks CLI, GitHub Actions, or GitLab.
• Monitor compute consumption, optimize cluster configurations, and enforce efficient resource use.

Skills

Python

PySpark

SQL

Git

Databricks

Delta Lake

Unity Catalog

DataOps

DevOps

Azure Cloud

GitHub Actions

Azure DevOps

GitLab CI/CD

Tools

Terraform

Databricks Asset Bundles

Databricks CLI

GitHub Actions

Azure DevOps

GitLab CI/CD

Job description

POSITION PURPOSE:

The Data & Analytics organization enables Bayer’s Corporate Functions—including Finance, Supply Chain & Logistics, HR, Procurement, Legal, and Sustainability to become more insight and data-driven. Our mission is to deliver a robust data foundation and FAIR—Findable, Accessible, Interoperable, and Reusable—data assets that improve business decision-making.

YOUR TASKS AND RESPONSIBILITIES:

• Design, build, and maintain scalable, reliable, and cost-efficient data pipeline architectures.


• Develop idempotent, automated ETL/ELT pipelines using Python, PySpark, Delta Lake, and MERGE INTO patterns.


• Drive automation across deployments, monitoring, incident remediation, and operational processes to improve efficiency, quality, and reliability.


• Implement robust DataOps practices, including data-contract enforcement, automated data-quality checks, and unit and integration testing.


• Establish fine-grained data governance, lineage tracking, and security access controls across environments using Unity Catalog.


• Define and manage Infrastructure as Code using Terraform and Azure DevOps for Databricks workspaces, compute clusters, serverless SQL warehouses, and storage integrations.


• Build and maintain automated CI/CD pipelines using Databricks Asset Bundles, Databricks CLI, GitHub Actions, Azure DevOps, or GitLab.


• Monitor compute consumption and costs, optimize cluster configurations, and implement cluster policies to promote efficient resource utilization.

WHO YOU ARE:

• At least five years of experience in data engineering, DataOps, DevOps, or a comparable technical role.


• Advanced hands‑on experience with Python, PySpark, SQL, Git, and Databricks.


• Strong experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, and Databricks Serverless SQL.


• Proven experience with Infrastructure as Code, preferably Terraform, and CI/CD tooling such as Databricks Asset Bundles, GitHub Actions, Azure DevOps, or GitLab.


• Experience implementing data‑quality controls, data contracts, automated testing, monitoring, and incident‑management practices.


• Strong knowledge of Azure cloud services, including security, networking, identity and access management, and storage integration. Experience with AWS or GCP is beneficial.

Ever feel burnt out by bureaucracy? Us too. That's why we're changing the way we work- for higher productivity, faster innovation, and better results. We call it Dynamic Shared Ownership (DSO). Learn more about what DSO will mean for you in your new role here https://www.bayer.com/enfstrategyfstrategy

Bayer is an equal opportunity employer that strongly values fairness and respect at work. We welcome applications from all individuals, regardless of race, religion, gender, age, physical characteristics, disability, sexual orientation etc. We are committed to treating all applicants fairly and avoiding discrimination.

Location: India : Karnataka : Bangalore

Division: Enabling Functions

Reference Code: 891948