Staff Data Engineer

bp; Pune District; Hybrid; INR 400,000 - 600,000; Full time

Benefits offered by this job

Hybrid work

Relocation assistance

Travel up to 10%

Job summary

bp is seeking a Staff Data Engineer in Pune to shape, design and deliver data and AI solutions at scale. You will lead technical design across data products and platforms, mentor teams, and drive end-to-end architecture from high-level direction to implementation.

This hands-on role emphasizes engineering excellence, with a focus on scalable data pipelines, secure solutions, and reusable patterns, while collaborating with cross-functional teams across bp.

Qualifications

• Bachelor’s degree in Computer Science or related field.
• 8+ years building scalable data infrastructure and data products.
• Experience producing HLD and LLD artefacts.
• Translate architecture into engineering specs for delivery teams.
• Strong software engineering practices: CI/CD, testing, monitoring.
• Python preferred; Java/Scala/C# also applicable.
• Expert SQL knowledge with large-scale distributed data systems.
• Experience with AI/ML in production, including LLM APIs.
• Strong stakeholder management and technical leadership without line management.
• Continuous learning mindset.

Responsibilities

• Translate high-level architecture into detailed, buildable designs.
• Set engineering standards across data pipelines and platforms.
• Design, build and maintain scalable data infrastructure and products.
• Integrate AI/ML capabilities into production systems.
• Provide technical direction across internal and partner teams.
• Ensure security, resilience, and operational readiness.

Skills

Data engineering

Architectural design

CI/CD

Mentoring

Stakeholder management

AI/ML integration

Python

SQL

Distributed systems

Education

Bachelor’s degree in Computer Science

Tools

Spark

Hadoop

Hive

LangChain

Weaviate

Hugging Face

Job description

Entity

Technology

Job Family Group

IT&S Group

Job Description

Staff Data Engineer

Individual Contributor, Grade G

About the role

The Staff Data Engineer plays a critical role in shaping, designing and delivering data and AI solutions that translate complex business needs into scalable, secure and reusable capabilities. Operating at staff level, this role provides senior technical leadership across data products, data platforms and AI-enabled solutions, with clear accountability for end-to-end system design, from high-level architecture through to low-level implementation design.

This is a hands-on engineering leadership role. The postholder independently identifies problems to solve, leads the technical design of solutions and acts as technical lead across one large-scale or multiple medium-scale delivery workstreams. A strong command of modern engineering patterns and practices is essential, as this role helps set the engineering standard for others to follow.

The role carries clear technical leadership accountability without line management responsibility. It guides delivery teams, influences internal and third-party engineering outcomes, and ensures solutions are well designed, maintainable, secure and fit for enterprise use. The postholder is also expected to mentor others and actively contribute to engineering excellence across the wider team.

You will work with

This role sits within the Finance Technology team and works closely with data engineers, data analysts, AI and software engineers, product managers, architects, cyber, platform and business stakeholder teams. It also partners with central technology and architecture teams to ensure solutions are designed for reuse, interoperability and future scalability, rather than being developed in isolation.

You will deliver

• End-to-end system design: Translate high-level architectural direction into detailed, buildable solution designs. This includes decomposing architectural patterns, such as event-driven ingestion, medallion lakehouse and RAG pipelines, into concrete engineering specifications including table schemas, transformation logic, interface contracts, retry and error-handling strategies, SLA thresholds and observability requirements. Produce HLD artefacts that communicate solution intent and platform choices, alongside LLD artefacts that delivery engineers can implement without ambiguity. Design outputs should be reviewable through architecture governance and usable as the baseline for technical assurance.
• Strong engineering practices and patterns: Set and reinforce standards for software design, code quality, testing, CI/CD, observability and documentation. Advocate for and apply established engineering patterns, including event-driven architecture, microservices, pipeline orchestration, RAG and layered data architecture, using sound judgement on when each pattern is appropriate. Apply these patterns effectively across data-intensive pipelines, products and applications.
• Scalable data infrastructure and product delivery: Design, build and maintain reliable data pipelines, data products and integration layers that move, process and serve data across the enterprise at scale.
• AI-enabled solution engineering: Integrate AI and machine learning capabilities into production-grade applications, including LLM-based solutions, retrieval-augmented generation, AI agents and unstructured data processing. Evaluate AI tooling pragmatically, prioritising measurable business value and sustainable delivery over experimentation for its own sake.
• Technology selection, reuse and scalable solution design: Exercise strong technical judgement in determining when to build custom capability, configure or extend existing platforms, adopt SaaS solutions, or use low-code/no-code approaches. Ensure decisions consider both the immediate requirement and broader enterprise value, balancing delivery speed, maintainability, cost and scalability. Identify opportunities to reuse existing capabilities, create common patterns and shared services, adopt platform-based solutions and actively reduce unnecessary duplication and technical debt.
• Technical governance and design assurance: Work with architecture, cyber, data governance and platform teams to ensure solutions meet enterprise standards for security, resilience, compliance, performance and operational readiness.
• Engineering leadership across internal and third-party teams: Provide technical direction and engineering leadership across internal teams, delivery partners and third-party suppliers. Set clear engineering expectations, challenge technical decisions where appropriate, and ensure delivered solutions meet required standards for quality, architecture, security, maintainability and long-term ownership.
• Operational excellence and lifecycle ownership: Design for supportability, observability, reliability and cost effectiveness throughout the solution lifecycle. Define service reliability expectations, SLA requirements and appropriate site-reliability engineering practices for systems owned or technically led by the role.
• Hands-on delivery and prototyping: Translate ambiguous problems into practical technology options, solution designs and working prototypes. Use modern engineering and AI-assisted development tools to accelerate design and build activities, test assumptions and validate ideas rapidly with users before committing to larger-scale implementation.
• Technical mentorship and engineering excellence: Mentor engineers across the team, contribute to data engineering learning and development, and actively participate in the evolution of engineering communities, standards, reusable patterns and practices.

What you will need to be successful

Essential

• Bachelor’s degree in Computer Science or a related field.
• 8+ years of hands-on experience designing, building, productionising and maintaining reliable, scalable data infrastructure and data products in complex environments.
• Demonstrable experience producing HLD and LLD artefacts for data or technology solutions, including solution architecture diagrams, data-flow designs, component specifications, API contracts, data models and operational runbooks.
• Ability to take architectural decisions and decompose them into engineering specifications that developers can build from without further clarification.
• Strong command of software engineering best practices, including technical design review, unit testing, CI/CD pipeline development, monitoring and alerting, code review and documentation.
• Development experience in one or more object-oriented programming languages, with Python preferred. Scala, Java or C# are also applicable.
• Expert SQL knowledge and experience working with large-scale distributed data systems.
• Experience integrating AI and machine learning capabilities into production systems, including LLMs through APIs, RAG patterns, vector search, embeddings or AI agents.
• Deep knowledge and hands-on experience across the full data lifecycle.
• Strong stakeholder management skills and the ability to lead through technical influence across internal teams and external partners.
• A continuous learning and improvement mindset.

Desired

• Experience with AI/ML frameworks such as LangChain, LlamaIndex, Hugging Face, Weaviate or similar ecosystems.
• Familiarity with agentic frameworks such as CrewAI, AutoGen or similar LLM orchestration toolkits.
• Experience with big data technologies such as Spark, Hadoop or Hive.
• Familiarity with cloud data platforms such as AWS, Azure or GCP, and infrastructure-as-code practices.
• Experience with data governance tools such as Alation or Collibra, and enterprise data management standards.
• No prior experience in the energy industry is required.

Why join bp

Our purpose is to deliver energy to the world, today and tomorrow. For over 100 years, bp has focused on discovering, developing, and producing oil and gas in the nations where we operate. We are one of the few companies globally that can provide governments and customers with an integrated energy offering. Delivering our strategy sustainably is fundamental to achieving our ambition to be a net zero company by 2050 or sooner! At bp, we support our people to learn and grow in a diverse and high-reaching environment. We are committed to fostering an inclusive environment in which everyone is respected and treated fairly.

Disclaimer

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform crucial job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation. Even though the job is advertised as full time, please contact the hiring manager or the recruiter as flexible working arrangements may be considered. We believe that our team is strengthened by diversity. We are committed to crafting an inclusive environment in which everyone is respected and treated fairly. There are many aspects of our employees’ lives that are important, so we offer benefits to enable your work to fit with your life. These benefits can include flexible working options, collaboration spaces in a modern office environment, and many others benefits.

What you can expect from us!

Our commitment to diversity, equity and inclusion: At bp, you could be part of Business Resource Groups (BRGs) which believe in the power of inclusion, deeper connections, and shared experiences. They provide a place for employees to learn and share knowledge, to connect, and to improve. The BRGs focus on and encourage talent engagement, development, and retention while creating a broadened sense of community and inclusion for bp employees. The groups cultivate leadership growth by involving employees in developmental opportunities they would not otherwise have access to. Formal and informal mentoring also helps employees develop their professional goals and connect with colleagues. The BRGs’ dedication to growth at all levels helps employees feel more engaged and energized. The BRGs break down barriers, so we can all do better together. Examples of BRGs include (but are not limited to): bpWIN (bp Women’s International Network) Pride Working parents PEN (Positively Ethnic Network) bpInclusia (bp’s Asian community) bpEnergía (bp’s Latin community)

Travel Requirement

Up to 10% travel should be expected with this role

Relocation Assistance

This role is eligible for relocation within country

Remote Type

This position is a hybrid of office/remote working

Skills

Agility core practices, Agility core practices, API and platform design, Big Data Engineering, Cloud Platforms, Collaboration, Communication, Configuration management and release, Continuous deployment and release, Creating a high performing team, Database Design, Data Engineering, Digital Project Management, Documentation and knowledge sharing, Emerging technology monitoring, Facilitation, Information Security, Mentoring, Metrics definition and instrumentation, NoSql data modelling, Problem Solving, Relational Data Modeling, Risk Management, Scripting, Secure development {+ 11 more}

.

Legal Disclaimer

We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status.Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process,please contact us.