Sr Data Engineer - ETRM

Shell; Bengaluru; Hybrid; INR 3,500,000 - 5,500,000; Full time

Benefits offered by this job

Flexible working hours

Remote/mobile working

Parental leave

Training and development

Job summary

Shell Energy Software Engineering & Analytics in Bengaluru seeks a Senior Data Engineer to design and optimize scalable data pipelines. You will work with Azure Data Factory, ADLS, Databricks, PySpark, Python and SQL to enable fast, reliable decisions.

Domain experience in Energy Trading (Endur/Nucleus) is highly valued. You will drive continuous improvement, collaborate with agile teams, and contribute to governance and data catalog initiatives while delivering high-quality data solutions

Qualifications

• 8–10 years of data engineering experience delivering enterprise-scale data solutions.
• Deep knowledge of Azure technologies and data platforms.
• Experience with Endur and Nucleus in Energy Trading domain.
• Strong SQL, PySpark, Python and data modeling skills.
• Hands-on DevOps, GitHub, CI/CD and release management experience.

Responsibilities

• Design, develop, and maintain scalable data pipelines for decision support.
• Build end-to-end data platforms using ADF, ADLS, Databricks and PySpark.
• Implement data modeling strategies for enterprise reporting and analytics.
• Collaborate in an Agile environment with cross-functional teams.
• Optimize data products and platform performance; ensure governance.

Skills

PySpark

Python

SQL

Azure

Databricks

MDM

Data Modeling

Agile

GitHub

CI/CD

Education

Bachelor's degree in Computer Science/Engineering

MCA or related

Tools

Azure Data Factory

Azure Data Lake Storage

Azure Databricks

Endur

Nucleus

Data Catalog

MDM

GitHub

Azure DevOps

Power BI

AI Foundry

Databricks Genie

Microsoft Fabric

Job description

Where you fit?

An exciting opportunity to work at the Trading & Supply- Shell Energy Software Engineering & Analytics organization. We are looking for a talented Senior Data Engineer with great interpersonal skills to join our team. Shell Trading and Supply is the world's largest Energy Trader. It combines a global network of trading companies, industry leading shipping and maritime capabilities, and an integrated network of supply and distribution activities to act as the central integrator for Royal Dutch Shell. Our trading capability, deep market knowledge, global portfolio and end-to-end integration within Shell creates opportunities to deliver value for our customers and for Shell across Upstream, Downstream, Integrated Gas and New Energies. Through our main trading offices in London, Houston, Singapore, Dubai and Rotterdam we trade crude oil, natural gas, LNG, electricity, refined products, chemical feedstocks and environmental products. With more than 125 distribution terminals and hundreds of outside supply points in around 25 countries, our supply and distribution infrastructure is well positioned to make deliveries around the world. Trading is at the centre of the Shell's Energy Transition and Decarbonisation plans, working closely with New Energies and the wider business to develop lower carbon solutions, driven through technology and data. If you would like to be at the centre of Shell's future, join our Trading & Supply Competency Centre and play a part in the delivery of our energy future.

What’s the role?

Senior Data Engineer is an individual with proven experience in adhering to Data Engineering best practices, delivering Data related optimal & performance driven solutions which helps business making quick decision while generating value. They are expected to have continuous improvement mindset, driving value-based process, product/application optimization.

What you’ll be doing?

• Design, develop, and maintain scalable, high-performance data engineering solutions that support business decision-making and generate measurable value.
• Build and optimize end-to-end data pipelines using Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Databricks, PySpark, Python, and SQL.
• Develop data engineering solutions for the Energy rading & Supply business, leveraging domain knowledge of ETRM ( Endur, and Nucleus).
• Implement data modeling strategies to support enterprise reporting, analytics, and operational requirements.
• Drive continuous improvement initiatives by optimizing data products, processes, applications, and platform performance.
• Apply performance tuning and optimization techniques using PySpark and SQL to improve data processing efficiency and scalability.
• Design and support enterprise data platforms, Master Data Management (MDM), Data Catalog, and analytics solutions.
• Collaborate with cross-functional teams in an Agile environment, ensuring adherence to data engineering best practices, quality standards, and governance requirements.
• Manage source control, CI/CD, and release management processes using GitHub, Azure DevOps, and related tools.
• Work with relational and non-relational databases to support data integration, transformation, and analytics initiatives.
• Partner with business and technical stakeholders to translate Energy Trading business requirements into scalable data solutions.
• Contribute to innovation initiatives, including the evaluation and adoption of emerging technologies and AI-driven solutions.

What we need from you?

• Bachelor's degree in Engineering, Computer Science, Information Technology, or a related field; MCA qualification is also acceptable.
• With at least 8 - 10 years of experience in Data Engineering, with a proven track record of delivering enterprise-scale data solutions.
• Strong experience in the Trading & Supply domain, including Endur and Nucleus.
• Deep understanding of the Energy Trading business domain and experience building data solutions that support trading operations and decision-making.
• Expertise in Microsoft Azure technologies, including:
  • Azure Data Factory (ADF)
  • Azure Data Lake Storage (ADLS)
  • Azure Databricks
  • Data Pipeline Development
  • Azure Fabric
• Strong hands-on experience with PySpark, Python, Advanced SQL and Data Modeling
• Experience working with both relational and non-relational database technologies.
• Proven expertise in SQL query optimization and PySpark performance tuning.
• Experience with source control and DevOps practices, including GitHub, Azure DevOps, build pipelines, and release management.
• Hands-on experience implementing enterprise data platforms, Master Data Management (MDM), Data Catalog, and analytics solutions.
• Experience working in Agile delivery environments and applying engineering best practices.
• Databricks Certified Associate certification required.
• Excellent verbal and written communication skills.

Preferred Qualifications

• Experience working with data products and platforms within the Energy sector.
• Knowledge of ETRM (Energy Trading and Risk Management) systems, particularly Endur, and its integration with enterprise energy applications.
• Experience implementing AI-powered solutions and platforms, such as AI Foundry and Databricks Genie.
• Hands-on experience with Microsoft Fabric and/or Power BI.
• Exposure to advanced analytics, data governance, and modern cloud-based data architectures.

What We Offer

You bring your skills and experience to Shell and in return you work with talented, committed people on one of the most important challenges facing our planet. You’ll have the opportunity to develop the skills you need to grow in an environment where we value honesty, integrity, and respect for one another. You’ll be able to balance your priorities as you become the best version of yourself.

• Progress as a person as we work on the energy transition together.
• Continuously grow the transferable skills you need to get ahead.
• Work at the forefront of technology, trends, and practices.
• Collaborate with experienced colleagues with unique expertise.
• Achieve your balance in a values-led culture that encourages you to be the best version of yourself.
• Benefit from flexible working hours, and the possibility of remote/mobile working.
• Perform at your best with a competitive starting salary and annual performance related salary increase – our pay and benefits packages are considered to be among the best in the world.
• Take advantage of paid parental leave, including for non-birthing parents.
• Join an organisation working to become one of the most diverse and inclusive in the world. We strongly encourage applicants of all genders, ages, ethnicities, cultures, abilities, sexual orientation, and life experiences to apply.
• Grow as you progress through diverse career opportunities in national and
• international teams.
• Gain access to a wide range of training and development programmes.