Global Data Engineering Lead - AI & Pipelines

bp; Greater London; Hybrid; GBP 110,000 - 150,000; Full time

Benefits offered by this job

Flexible working options

Retirement benefits

Job summary

BP is seeking a Senior Enterprise Tech Engineer to establish and lead the Data Engineering squad within the TA&I organisation of the T&S entity. You will design and implement enterprise-scale data pipelines, mentor 8–10 engineers, and drive AI-enabled data capabilities across global environments.

You will leverage Python, SQL, Airflow, and AWS to deliver robust data platforms, while championing automation, DevOps practices, and clear technical documentation in a dynamic trading-focused setting.

Qualifications

• Bachelor/Master’s degree or equivalent in CS/Engineering or numerate field.
• 10 years as a Data Engineer with mentoring leadership.
• Experience building market data solutions in trading/financial services.
• Expertise in Python (Pandas, Selenium/Beautiful Soup, Requests) and ML libs.
• Generative AI techniques including agents, RAG, AI SDLC, context engineering.
• Mastery of SQL and PostgreSQL; data modelling and design.
• Extensive experience with large-scale data pipelines and time-series data.
• Apache Airflow for data orchestration at scale.
• AWS cloud services (S3, Redshift, EC2, Lambda).
• SDLC design using Git and ADO/Jenkins pipelines.
• Linux environment and Git proficiency.
• Excellent communication and global collaboration skills.
• Desire to build high-performing teams and foster learning.

Responsibilities

• Establish and lead a global data engineering squad and define technical direction.
• Design and implement scalable data pipelines using Python, SQL and Airflow.
• Lead ETL of large datasets from diverse sources including time-series data.
• Leverage Generative AI to deliver novel capabilities for trading org.
• Utilize AWS to build and manage secure, cost-effective pipelines.
• Mentor 8–10 data engineers and promote knowledge sharing.
• Collaborate across borders in a global, cross-functional team.
• Drive Agile adoption (Kanban/Scrum) and manage workloads via ADO/CI-CD.
• Create data visualisations with Tableau/Power BI for diverse audiences.
• Maintain clear technical documentation for pipelines and procedures.
• Foster continuous improvement through sprints, RCAs and design reviews.

Skills

Python

SQL

Apache Airflow

AWS

Generative AI

CI/CD/DevOps

Data pipelines

Leadership

Time series data

Education

Bachelor’s or Master’s degree in CS/Engineering

Tools

Databricks

Tableau/Power BI

Git/Azure DevOps/Jenkins

Job description

BP is seeking a Senior Enterprise Tech Engineer to establish and lead the Data Engineering squad within the TA&I organisation of the T&S entity. You will design and implement enterprise-scale data pipelines, mentor 8–10 engineers, and drive AI-enabled data capabilities across global environments.

You will leverage Python, SQL, Airflow, and AWS to deliver robust data platforms, while championing automation, DevOps practices, and clear technical documentation in a dynamic trading-focused setting.