Lead Data Engineer

Elanco; United States; Hybrid; USD 41,000 - 72,000; Full time

Job summary

Elanco is seeking a Lead Data Engineer in Bangalore, India with 6+ years of experience to modernize ERP data architecture. You will build near real-time SAP data replication into Google BigQuery and integrate with Databricks for AI/ML and global reporting.

You will own production pipelines, drive efficiencies, and mentor junior engineers while collaborating with analysts and data scientists to translate requirements into robust technical stories.

Qualifications

• Bachelor's degree or equivalent in CS/SE.
• 6+ years delivering enterprise data solutions.
• Experience with Google BigQuery and Databricks required.

Responsibilities

• Own end-to-end data pipelines across SAP datasets.
• Mentor engineers on tooling and testing patterns.
• Own production incidents and implement permanent fixes.
• Collaborate with analysts/scientists to translate requirements into technical stories.
• Promote Elanco data standards and code reviews.
• Monitor pipelines and optimize performance/memory usage.
• Design data quality checks and CI/CD tests.
• Share knowledge and contribute to the data engineering community.

Skills

BigQuery

Databricks PySpark

SQL

Python

Spark

DevSecOps

CI/CD

TDD

Education

Bachelor's degree in Computer Science or related

Tools

Git

GitHub

VS Code

SonarQube

Copilot

BigQuery Studio

Databricks

Job description

At Elanco (NYSE: ELAN) – it all starts with animals!

As a global leader in animal health, we are dedicated to innovation and delivering products and services to prevent and treat disease in farm animals and pets. At Elanco, we are driven by our vision of Food and Companionship Enriching Life and our purpose – all to Go Beyond for Animals, Customers, Society and Our People.

At Elanco, we pride ourselves on fostering a diverse and inclusive work environment. We believe that diversity is the driving force behind innovation, creativity, and overall business success. Here, you’ll be part of a company that values and champions new ways of thinking, work with dynamic individuals, and acquire new skills and experiences that will propel your career to new heights.

Making animals’ lives better makes life better – join our team today!

Your Role:

Lead Data Engineer

Data Engineering at Elanco delivers products and thought leadership that transform how the organization leverages data. The ERP Enterprise Data Product (EEDP) team is a critical pillar of this strategy, responsible for modernizing our complex ERP data landscape. EEDP provides a robust, near real-time replication of SAP S/4HANA data into Google BigQuery, which is then integrated into our Enterprise Lakehouse (Databricks) to power advanced AI/ML, conversational analytics, and global reporting.

Your Responsibilities

• Complex Pipeline Ownership: Independently design and implement complex end-to-end data pipelines spanning multiple sources and large SAP datasets (Configuration, Master Data, Transactional). Anticipate scaling needs, data skew, and growth using patterns like partitioning, clustering, and incremental backfills.
• Technical Mentorship:Act as the primary gatekeeper for technical standards; ensure junior engineers are proficient in foundational tooling (e.g., Git branching, environment setup) and consistently apply testing patterns.
• Production Ownership:Own routine production incidents in familiar pipelines (schema drift, late files, logic errors) and deliver permanent fixes by updating code, tests, and documentation.
• Stakeholder Collaboration:Work directly with analysts and data scientists to refine requirements into concrete technical stories, proactively surfacing edge cases such as late-arriving data or schema changes.
• Standards Advocacy:Ensure the adoption of Elanco’s data standards and foundational tools (GitHub, VS Code, SonarQube, Copilot) across the team, using code reviews as a teaching tool.
• Efficiency & Optimization:Monitor key pipelines using logs and dashboards; identify and fix performance or memory issues using query plans and Spark UI without assistance.
• Quality Assurance:Design and implement meaningful data quality checks (referential integrity, distribution checks) and ensure tests run in CI/CD schedules.
• Continuous Improvement:Proactively identify and fix knowledge silos by creating reusable assets and documentation to ensure peers can follow established patterns.
• Community Contribution:Contribute to the Data Engineering community across Elanco to inspire, engage, and ignite innovation.
• Learning Mindset:Embrace and demonstrate a growth and sharing mindset, leveraging AI to accelerate intermediate tasks and tackle complex engineering challenges.

What You Need to Succeed (minimum qualifications)

• Bachelor’s degree in computer science, Software Engineering, or equivalent professional experience.
• 6+ years of experience engineering and delivering enterprise scale data solutions.
• Proven experience with Google BigQuery (BigQuery Studio) and Databricks (PySpark/Delta Lake) is required.
• Familiarity with SAP data structures (S/4HANA), SAP BW and experience with replication tools like SAP SLT (System Landscape Transformation Server) is highly preferred.
• Experience mentoring junior engineers, stewarding a practitioner community, and/or ensuring a team’s adherence to technical standards.

What will give you a competitive edge (preferred qualifications)

• Proven ability to independently deliver moderately complex data projects within a defined design.
• Strong proficiency in SQL (multi-join, window functions), Python, and Spark for building and maintaining data products.
• Experience working with modern data architectures, including Lakehouse (Bronze/Silver/Gold), star schemas, and data contracts.
• Experience working within a DevSecOps culture, including Git, CI/CD, and Test-Driven Development (TDD).
• Familiarity with data quality frameworks, data governance, and cost/efficiency considerations in cloud environments.
• Experience working in diverse global landscapes (business, technology, regulatory, and geography).
• Excellent interpersonal and communication skills; proven ability to turn rough requirements into clear, testable specifications.

Additional Information

• Travel:0%
• Location: IN, Bangalore - Hybrid Work Environment

Don’t meet every single requirement?Studies have shown underrecognized groups are less likely to apply to jobs unless they meet every single qualification. At Elanco we are dedicated to building a diverse and inclusive work environment. If you think you might be a good fit for a role but don't necessarily meet every requirement, we encourage you to apply. You may be the right candidate for this role or other roles!

Elanco is an EEO/Affirmative Action Employer and does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status

Elanco may use automated tools, including AI, to support parts of our recruitment process, such as reviewing applications against job‑related criteria and/or transferrable skills. These tools help ensure a consistent, structured evaluation, but they do not make hiring decisions. All decisions involve a human reviewer. For more information on how we handle personal data, please see our Elanco Workforce Privacy Notice.