Senior Data Engineer

bp; Pune District; Hybrid; INR 2,600,000 - 4,200,000; Full time

Benefits offered by this job

Pension

Private healthcare

Flexible working

Learning & career growth support

Relocation assistance

Job summary

bp is seeking a Senior Data Engineer for its bp pulse data platforms in a hybrid model. You will design and run scalable data pipelines on Databricks and AWS, model data for analysis, and ensure governance and privacy compliance while mentoring peers to raise technical standards.

You will collaborate across the business to translate needs into dependable data products, build CI/CD and infrastructure as code, and maintain reliability with monitoring and incident response.

Qualifications

• Hands-on experience building and running reliable, scalable data pipelines.
• Strong Databricks and AWS data services knowledge.
• Proficient in Python and advanced SQL with production data modelling.

Responsibilities

• Design, build and run reliable data pipelines on Databricks and AWS.
• Model data for analysis and ensure governance and accuracy.
• Develop secure, tested software for data integration with privacy compliance.
• Enforce engineering practices: design reviews, testing, source control, docs.
• Build CI/CD pipelines and manage infrastructure as code.
• Monitor, alert and support pipelines; improve reliability across services.
• Mentor engineers and raise team quality through leadership by example.
• Collaborate with business teams to translate needs into data products.

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Workflows

AWS

Python

SQL

Data modelling

Infrastructure as code

CI/CD

Mentoring

Technical influence

Curiosity / continuous improvement

Education

Bachelors in CS or related field

Tools

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Workflows

Terraform / IaC tooling

Job description

Entity:

Technology

Job Family Group:

IT&S Group

Job Description:

Let me tell you about the role

Our data engineers build and run the data pipelines and platforms that bp pulse relies on every day. The team builds on Databricks and AWS.

As a Senior Data Engineer, you will design and build data pipelines, model data for analysis, and keep our platforms dependable and well-governed. It’s a hands-on, senior role: setting a high engineering standard, mentoring other engineers, and acting as an informal technical lead within the squad leading through experience rather than a formal management line.

What you will deliver

• Design, build and run reliable, scalable, distributed data pipelines on Databricks and AWS.
• Model data for analysis, keeping it accurate, well-governed and easy to use.
• Write secure, well-tested software to build, integrate and quality-assure data across bp pulse, meeting privacy and compliance needs.
• Follow and encourage good engineering practice: technical design and review, testing, code review, source control and documentation.
• Build and improve CI/CD pipelines and manage infrastructure as code.
• Look after the pipelines and services the team builds monitoring, alerting and business-hours support and keep improving their reliability.
• Mentor other engineers and share knowledge to lift the quality of the team’s work.
• Work with people across the business to understand what they need and turn it into dependable data products.

What you will need to be successful

Essential

• Strong, hands-on experience building and running reliable, scalable, distributed data pipelines and data products in complex environments.
• Hands-on experience with Databricks, PySpark, Spark SQL, Delta Lake, Unity Catalog and Workflows (or similar lakehouse experience, with a real appetite to grow on Databricks).
• Experience with AWS data and serverless services (e.g. Lambda, Glue, S3, Redshift, Kinesis, SNS/SQS).
• Strong Python and advanced SQL.
• Good data modelling skills, applied in production.
• Experience with infrastructure as code and CI/CD.
• Experience mentoring engineers and helping raise the quality bar.
• Comfortable working with people across the business and shaping work through technical influence.
• A curious, continuous-improvement mindset.
• A degree in computer science or a related field, or equivalent knowledge and experience.

Desired

• Experience with dbt or Apache Airflow.
• Familiarity with open table formats such as Delta, Apache Hudi or Apache Iceberg.
• Experience with Terraform or another infrastructure-as-code tool.
• No prior experience in the energy industry needed.

What we offer

In return, bp offers a competitive salary and bonus, a strong pension, private healthcare, flexible working and real support for your learning and career growth. [P&C to confirm country-specific benefits.]

About bp

Our purpose is to deliver energy to the world, today and tomorrow. For over 100 years, bp has focused on discovering, developing and producing oil and gas in the nations where we operate. We are one of the few companies globally that can provide governments and customers with an integrated energy offering. Delivering our strategy sustainably is fundamental to achieving our ambition to be a net zero company by 2050 or sooner!

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform crucial job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

Travel Requirement

No travel is expected with this role

Relocation Assistance:

This role is eligible for relocation within country

Remote Type:

This position is a hybrid of office/remote working

Skills:

Cloud Platforms, Cloud Platforms, Collaboration, Communication, Configuration management and release, Continuous deployment and release, Creating a high performing team, Database Design, Digital Project Management, Documentation and knowledge sharing, Emerging technology monitoring, Facilitation, Information Security, Mentoring, Metrics definition and instrumentation, NoSql data modelling, Problem Solving, Relational Data Modeling, Risk Management, Scripting, Secure development, Service operations and resiliency, Software Design and Development, Solution Architecture, Source control and code management {+ 5 more}

.

Legal Disclaimer:

We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status.Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process,pleasecontact us.

If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.