Data Engineer_Python,SQL,Databricks,AWS

bp; Pune District; Híbrido; INR 1 600 000 - 2 800 000; Tempo integral

Resumo da oferta

bp is seeking an Enterprise Tech Engineer - Data Engineering in Pune to design, build, and support scalable data pipelines for BI and analytics platforms. You will ensure data is accurate, timely, secure, and easy to use across systems, supporting trading analytics and insights with reliable data products.

Applicants should have 3+ years of data engineering experience, strong Python/Pandas, advanced SQL, and hands-on Databricks/Spark, plus AWS data services exposure.

Qualificações

• 3+ years of data engineering experience in an enterprise environment.
• Strong Python and Pandas skills, including HTTP/API integration.
• Advanced SQL and solid understanding of relational data modelling.
• Hands-on Databricks and distributed processing experience; Spark/PySpark and Delta Lake are strongly preferred.

Responsabilidades

• Design, build, and support scalable data pipelines for business intelligence and analytics platforms.
• Ensure data is accurate, timely, secure, and easy to use across systems.
• Develop cloud-based data solutions using AWS services such as S3, EC2, IAM, Glue, and CloudWatch where appropriate.
• Optimise, monitor, and troubleshoot pipelines for performance, cost, resilience, and scalability.

Conhecimentos

Python

Pandas

SQL

Apache Airflow

Databricks

Spark/PySpark

AWS

Tableau/Power BI

Git/CI/CD

Docker

Formação académica

Bachelor’s or Master’s degree in Computer Science or Engineering

Ferramentas

Databricks

Airflow

Spark/PySpark

Kafka

Terraform/Jenkins

Azure DevOps

Docker

Descrição da oferta de emprego

Enterprise Tech Engineer - Data Engineering

Digital Trading Analytics | Trading Analytics & Insights (TA&I)

At bp, our people are our most valuable asset. The Technology DS&C, People, Culture & Communications (PC&C) function fosters a diverse, inclusive culture where everybody can thrive. As we transition from an oil company to an integrated energy company, we are embarking on a major transformation to be more competitive, responsive, and customer-focused.

We're investing in key locations such as India, Hungary, Malaysia, and Brazil, offering an exciting but challenging opportunity to shape a fast-moving PC&C function, building teams and structures and driving continuous improvement.

We're looking for driven, ambitious enthusiasts who thrive in fast-paced environments and are passionate about people. If you're ready to build something transformative, this is the place for you.

The Global Solution Owner Automation and AI is responsible for owning the products, architecture, vendor relationships, and compliance globally.

You will work with

You will join Digital Trading Analytics and support the Trading Analytics & Insights (TA&I) organisation. We are a global, high-performing team that works closely with traders, trading analysts, business analysts, and engineers to turn data into useful insight for energy trading.

You will work in an Agile delivery environment using Kanban and Scrum practices. Azure DevOps is used to plan, track, and deliver work.

About the role

As an Enterprise Tech Engineer - Data Engineering, you will design, build, and support scalable data pipelines for business intelligence and analytics platforms. You will ensure data is accurate, timely, secure, and easy to use across systems. Your work will directly support trading operations through reliable, high-quality data products.

You will also use approved AI-enabled engineering tools responsibly to improve coding speed, quality, documentation, and problem solving. This is not an AI research role; it requires practical AI fluency and sound engineering judgement.

What you will deliver

• Architect, design, and implement robust data pipelines using Python, Pandas, SQL, Apache Airflow, and Databricks.


• Build and maintain ETL/ELT processes that ingest data from APIs, relational databases, event streams, and flat files.


• Develop cloud-based data solutions using AWS services such as S3, EC2, EKS, IAM, Glue, and CloudWatch where appropriate.


• Optimise, monitor, and troubleshoot pipelines for performance, cost, resilience, and scalability.


• Use distributed processing technologies such as Spark/PySpark and Delta Lake for large-scale workloads.


• Implement data quality checks, lineage, observability, and clear operational runbooks.


• Create clear data visualisations and technical communication for both technical and non-technical audiences.


• Work with multi-functional partners to translate trading and business needs into fit-for-purpose data solutions.


• Apply Agile practices, version control, code review, automated testing, and release subject area.

AI-enabled engineering and productivity

You will use approved AI tools to accelerate delivery while protecting company data, validating output, and retaining accountability for production decisions.

• Understand practical AI fundamentals, including how LLMs generate responses, common limitations, hallucinations, and privacy risks.


• Use coding assistants and AI tools effectively for code drafts, tests, documentation, SQL review, debugging, and knowledge discovery.


• Write clear prompts: state the goal, constraints, inputs, expected output, and acceptance checks.


• Create useful context for AI tools using relevant schemas, examples, business rules, and bounded source material without exposing critical information.


• Understand RAG fundamentals: document retrieval, grounding, citations, evaluation, and access control.


• Evaluate AI-generated work through testing, peer review, source checks, and security review before production use.


• Keep current with approved emerging AI capabilities and propose practical uses that improve productivity or data quality.

What you will need to be successful

• 3+ years of data engineering experience in an enterprise environment.


• Strong Python and Pandas skills, including HTTP/API integration (for example, Requests). Use web automation only where it is approved, lawful, and appropriate for the source.


• Advanced SQL and solid understanding of relational data modelling and databases.


• Proven ETL/ELT and pipeline design experience, including hands-on Apache Airflow.


• Hands-on Databricks and distributed processing experience; Spark/PySpark and Delta Lake are strongly preferred.


• Experience building and operating data solutions on AWS, including S3 and relevant managed data services; familiarity with IAM is expected.


• Experience with EKS is helpful where workloads run on Kubernetes.


• Experience with messaging or streaming systems such as Kafka; familiarity with event-driven data patterns is helpful.


• Solid understanding of Linux, Git, pull-request workflows, and CI/CD. Azure DevOps experience is helpful.


• Familiarity with Docker, infrastructure as code (for example Terraform, Jenkins), and secure cloud deployment practices.


• Experience with Tableau, Power BI, or similar visualisation tools.


• Understanding of secure engineering standards and documentation practices.


• Basic practical knowledge of LLMs, RAG, prompt design, context design, and safe use of AI developer tools.


• Strong analytical, problem-solving, communication, collaboration, and organisational skills.


• A responsible, self-motivated, adaptable approach and a genuine interest in automation.


• Experience in trading, financial services, energy, or another regulated data-intensive domain is advantageous.


• A Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.

Why bp?

At bp, we support our people to learn and grow in a diverse and challenging environment. We believe that our team is strengthened by diversity. We are committed to crafting an inclusive environment in which everyone is respected and treated fairly. Diversity Statement: At bp, we provide an excellent environment and benefits such as an open and inclusive culture, a great work-life balance, tremendous learning and development opportunities to craft your career path, life and health insurance, medical care package and many others!

Diversity sits at the heart of our company and as an equal opportunity employer, we stay true to our mission by ensuring that our place can be anyone's place. We do not discriminate based on race, religion, colour, national origin, gender and gender identity, sexual orientation, age, marital status, veteran status or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application and interview process, to perform essential job functions, and to receive other benefits and privileges of employment.