Lead Data Engineer

Elanco; Bengaluru; On-site; INR 3,500,000 - 6,500,000; Full time

Job summary

Elanco Tiergesundheit AG is seeking a Lead Data Engineer to own end-to-end SAP data pipelines, including SAP S/4HANA replication to BigQuery and Databricks. The role emphasizes quality, mentorship, and collaboration with analysts and data scientists to deliver scalable data products.

You will promote data standards, monitor performance, and drive continuous improvement in a hybrid Bangalore work environment. A strong background in SQL, Python, Spark, and cloud data architectures is essential.

Qualifications

• Bachelor’s degree in computer science, Software Engineering, or equivalent professional experience.

Responsibilities

• Own complex end-to-end data pipelines across multiple SAP datasets and ensure scalability and reliability.
• Mentor junior engineers and enforce technical standards across the team.
• Own production incidents and deliver permanent fixes with updated tests and docs.
• Collaborate with analysts and data scientists to translate requirements into technical stories.
• Champion Elanco data standards and tooling across the team.
• Monitor pipelines and optimize performance using Spark UI and query plans.
• Design data quality checks and CI/CD tested pipelines.
• Create reusable assets and documentation to reduce knowledge silos.
• Contribute to the Data Engineering community within Elanco.

Skills

BigQuery

Databricks

SQL

Python

Spark

Git

CI/CD

TDD

S/4HANA

Communication

Mentoring

Education

Bachelor’s degree in computer science or equivalent

Tools

Git

CI/CD

Test-Driven Development

Job description

Your Role

Lead Data Engineer Data Engineering at Elanco delivers products and thought leadership that transform how the organization leverages data. The ERP Enterprise Data Product (EEDP) team is a critical pillar of this strategy, responsible for modernizing our complex ERP data landscape. EEDP provides a robust, near real-time replication of SAP S/4HANA data into Google BigQuery, which is then integrated into our Enterprise Lakehouse (Databricks) to power advanced AI/ML, conversational analytics, and global reporting.

Your Responsibilities

• Complex Pipeline Ownership: Independently design and implement complex end-to-end data pipelines spanning multiple sources and large SAP datasets (Configuration, Master Data, Transactional). Anticipate scaling needs, data skew, and growth using patterns like partitioning, clustering, and incremental backfills.
• Technical Mentorship: Act as the primary gatekeeper for technical standards; ensure junior engineers are proficient in foundational tooling (e.g., Git branching, environment setup) and consistently apply testing patterns.
• Production Ownership: Own routine production incidents in familiar pipelines (schema drift, late files, logic errors) and deliver permanent fixes by updating code, tests, and documentation.
• Stakeholder Collaboration: Work directly with analysts and data scientists to refine requirements into concrete technical stories, proactively surfacing edge cases such as late-arriving data or schema changes.
• Standards Advocacy: Ensure the adoption of Elanco’s data standards and foundational tools (GitHub, VS Code, SonarQube, Copilot) across the team, using code reviews as a teaching tool.
• Efficiency & Optimization: Monitor key pipelines using logs and dashboards; identify and fix performance or memory issues using query plans and Spark UI without assistance.
• Quality Assurance: Design and implement meaningful data quality checks (referential integrity, distribution checks) and ensure tests run in CI/CD schedules.
• Continuous Improvement: Proactively identify and fix knowledge silos by creating reusable assets and documentation to ensure peers can follow established patterns.
• Community Contribution: Contribute to the Data Engineering community across Elanco to inspire, engage, and ignite innovation.
• Learning Mindset: Embrace and demonstrate a growth and sharing mindset, leveraging AI to accelerate intermediate tasks and tackle complex engineering challenges.

What You Need to Succeed (minimum qualifications)

• Bachelor’s degree in computer science, Software Engineering, or equivalent professional experience.
• 6+ years of experience engineering and delivering enterprise scale data solutions.
• Proven experience with Google BigQuery (BigQuery Studio) and Databricks (PySpark/Delta Lake) is required.
• Familiarity with SAP data structures (S/4HANA), SAP BW and experience with replication tools like SAP SLT (System Landscape Transformation Server) is highly preferred.
• Experience mentoring junior engineers, stewarding a practitioner community, and/or ensuring a team’s adherence to technical standards.

What will give you a competitive edge (preferred qualifications)

• Proven ability to independently deliver moderately complex data projects within a defined design.
• Strong proficiency in SQL (multi-join, window functions), Python, and Spark for building and maintaining data products.
• Experience working with modern data architectures, including Lakehouse (Bronze/Silver/Gold), star schemas, and data contracts.
• Experience working within a DevSecOps culture, including Git, CI/CD, and Test-Driven Development (TDD).
• Familiarity with data quality frameworks, data governance, and cost/efficiency considerations in cloud environments.
• Experience working in diverse global landscapes (business, technology, regulatory, and geography).
• Excellent interpersonal and communication skills; proven ability to turn rough requirements into clear, testable specifications.

Additional Information

Travel: 0%

Location: IN, Bangalore - Hybrid Work Environment

Don’t meet every single requirement? Studies have shown underrecognized groups are less likely to apply to jobs unless they meet every single qualification.

Elanco is an EEO/Affirmative Action Employer and does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

Elanco may use automated tools, including AI, to support parts of our recruitment process, such as reviewing applications against job‑related criteria and/or transferrable skills. These tools help ensure a consistent, structured evaluation, but they do not make hiring decisions. All decisions involve a human reviewer. For more information on how we handle personal data, please see our Elanco Workforce Privacy Notice.

Elanco Animal Health Incorporated (NYSE: ELAN) is a global leader in animal health dedicated to innovating and delivering products and services to prevent and treat disease in farm animals and pets, creating value for farmers, pet owners, veterinarians, stakeholders, and society as a whole. With nearly 70 years of animal health heritage, we are committed to helping our customers improve the health of animals in their care, while also making a meaningful impact on our local and global communities. At Elanco, we are driven by our vision of Food and Companionship Enriching life and our Elanco Healthy Purpose CSR framework – all to advance the health of animals, people and the planet. Learn more at www.elanco.com.