Lead Data Engineer

Arup; Hyderabad; On-site; INR 2,600,000 - 3,800,000; Full time

Benefits offered by this job

Profit sharing

Learning opportunities

Career development

Job summary

Arup is seeking an experienced Lead Data Engineer to shape, build and operate scalable data solutions on Databricks. This hands-on leadership role requires setting technical direction, mentoring engineers and delivering robust data pipelines, Lakehouse solutions and analytics-ready data products.

You will collaborate with data teams and stakeholders to translate complex requirements into reliable services, promote best practices in governance, security and performance, and drive continuous

Qualifications

• Strong production experience with Databricks, Spark, PySpark, SQL and Delta Lake.
• Experience designing scalable data pipelines and Lakehouse solutions in the cloud.
• Knowledge of Unity Catalog, data governance, security and access control patterns.
• Confidence leading, mentoring and developing data engineers.
• A passion for engineering excellence, automation, reliability and continuous improvement.

Responsibilities

• Lead and mentor a team of data engineers, setting technical direction.
• Design, build and operate scalable data pipelines and Lakehouse solutions on Databricks.
• Define standards for ingestion, transformation, testing, deployment and monitoring.
• Collaborate with cross-functional teams to translate requirements into data services.
• Drive reusable frameworks, automation and modern data engineering practices.
• Embed data quality, governance and compliance across pipelines.
• Review designs, code and deployment approaches for reliability.
• Optimize Databricks workloads for performance and cost.
• Support Agile delivery, planning and roadmap development.

Skills

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Lakehouse

Data pipelines

Mentoring

Cloud-native data services

Unity Catalog

Tools

Databricks tooling

Job description

Arup’s purpose, shared values and collaborative approach has set us apart for over 80 years, guiding how we shape a better world.

We are looking for an experienced Lead Data Engineer to help shape, build and operate scalable data solutions on the Databricks platform. This is a hands‑on leadership role for someone who can set technical direction, mentor engineers and deliver robust, secure and high‑performing data pipelines, Lakehouse solutions and analytics‑ready data products.

You will work closely with data engineers, analysts, data scientists, architects, product owners and business stakeholders to translate complex requirements into practical, reliable and scalable data services. You will bring strong technical judgement, a passion for engineering excellence and the ability to promote best practice across data engineering, governance and platform operations.

At Arup, you belong to an extraordinary collective – in which we encourage individuality to thrive. Our strength comes from how we respect, share and connect our diverse experiences, perspectives and ideas. You will have the opportunity do socially useful work that has meaning – to Arup, to your career, to our members and to the clients and communities we serve.

• Lead, mentor and support a team of data engineers, setting technical direction and encouraging engineering best practice.
• Design, build and operate scalable data pipelines and Lakehouse solutions using Databricks, Apache Spark, Delta Lake and cloud‑native services.
• Define standards for ingestion, transformation, orchestration, testing, deployment, observability and monitoring.
• Collaborate with cross‑functional teams to turn business and technical requirements into high‑quality data solutions.
• Champion reusable frameworks, automation, medallion architecture patterns and modern data engineering practices.
• Embed data quality, security, governance and compliance controls across pipelines and data products.
• Review solution designs, code and deployment approaches to maintain consistency, resilience and operational reliability.
• Optimise Databricks workloads for performance, scalability and cost efficiency.
• Support Agile delivery, sprint planning, estimation, continuous improvement and technical roadmap development.

Is this role right for you?

• Strong production experience in Databricks, Spark, PySpark, SQL and Delta Lake.
• Experience designing scalable data pipelines, Lakehouse solutions and cloud‑native data services.
• Knowledge of Unity Catalog, data governance, security and access control patterns.
• Confidence leading, mentoring and developing data engineers.
• A passion for engineering excellence, automation, reliability and continuous improvement.

What we offer you

At Arup, we care about each member’s success, so we can grow together.

Guided by our values, we provide an attractive total reward package that recognises the contribution of each of our members to our shared success. As well as competitive, fair and equitable pay, we offer a career in which all of our members can belong, grow and thrive – through benefits that support health and wellbeing, a wide range of learning opportunities and many possibilities to have an impact through the work they do.

We are owned in trust on behalf of our members, giving us the freedom, with personal responsibility, to set our own direction and choose work that aligns with our purpose and adds to Arup’s legacy. Our members collaborate on ambitious projects to deliver remarkable outcomes for our clients and communities. Profit Share is a key part of our reward, enabling members to share in the results of our collective efforts.

Different people, shared values

Arup is an equal opportunity employer that actively promotes and nurtures a diverse and inclusive workforce. We welcome applications from individuals of all backgrounds, regardless of age (within legal limits), gender identity or expression, marital status, disability, neurotype or mental health, race or ethnicity, faith or belief, sexual orientation, socioeconomic background, and whether you’re pregnant or on family leave. We are an open environment that embraces diverse experiences, perspectives, and ideas – this drives our excellence.

Guided by our values and alignment with the UN Sustainable Development Goals, we create and contribute to equitable spaces and systems, while cultivating a sense of belonging for all. Our internal employee networks support our inclusive culture: from race, ethnicity and cross‑cultural working to gender equity and LGBTQ+ and disability inclusion – we aim to create a space for you to express yourself and make a positive difference. Discover more about life at Arup at Careers - Your Life at Arup .

We are committed to making our recruitment process and workplaces accessible to all candidates. Please let us know if you need any assistance or reasonable adjustments throughout your application or interview process, and/or to perform the essential functions of the role. We will do everything we can to support you.

An environment that will give you freedom to design with a purpose, focus on your skills to deliver quality work to our clients, opportunity to learn and share knowledge openly. You will have the opportunity to utilise your technical expertise to improve the lives of the most vulnerable in the communities in which we live and work through Arup's community engagement programme.

You'll have access to lots of learning opportunities and ongoing training through Arup university and other learning programmes.

Did you know: Our Digital Technology team are proud prior winners of the IT Team of the Year Award at the Chartered Institute of IT’s UK IT Industry Awards.