Lead Data Engineer

Anson McCade; Greater London; On-site; GBP 110,000 - 150,000; Full time

Job summary

Anson McCade Pty in London is seeking a Lead Data Engineer to join a growing technology consultancy. You will design and lead large-scale data platforms, while remaining hands-on and guiding engineers across projects.

You will work with clients, solution architects and security teams to ensure robust, scalable data solutions that meet regulatory requirements and business needs.

Qualifications

• Proven experience leading software teams on data-intensive systems.
• Strong development experience in Java, Scala or Python.
• Extensive Palantir experience in enterprise data solutions.
• Experience with cloud platforms (AWS/Azure/GCP) or Databricks.
• Expert SQL skills for analytical data processing.
• Knowledge of distributed data stores and large-scale processing.

Responsibilities

• Lead design, development and implementation of data-intensive systems.
• Hands-on software engineering using open source and data platforms.
• Establish coding standards and coaching across teams.
• Define robust ETL/ELT pipelines with performance and scalability.
• Own testing standards and performance optimisation.
• Lead due diligence and integration of software products.
• Collaborate with operations to ensure deployment readiness.
• Advise clients and stakeholders on approaches and estimates.
• Contribute to proposals and pre-sales activities.
• Manage and develop a small engineering team.

Skills

Team leadership

Stakeholder communication

Mentoring

Agile delivery

Tools

Palantir

AWS

Azure

GCP

Databricks

Job description

An exciting opportunity has arisen for an experienced Lead Data Engineer to join a growing technology consultancy, leading the design and development of large scale, data intensive solutions for major public and private sector organisations.

The successful candidate will combine strong hands on software engineering expertise with technical leadership, taking ownership of complex data engineering components and ensuring solutions are robust, scalable, secure and aligned with business and user requirements.

Working within agile delivery teams, they will lead engineers, establish development standards and drive best practice across the design, development, testing and deployment of modern data platforms.

Key Responsibilities

• Lead the design, development and implementation of data intensive systems, taking ownership of complete components and subsystems from initial design through to coding, testing and defect resolution.
• Remain hands on with software engineering, using open source technologies and commercial data platforms to build scalable data processing and data persistence solutions.
• Establish and enforce software development standards, design patterns and coding practices, coaching engineers to maintain consistency and quality across delivery teams.
• Work closely with solution architects to define and deliver robust ETL and ELT data pipelines, with particular focus on performance, resilience, scalability and non functional requirements.
• Take ownership of unit and integration testing standards, ensuring software quality through effective testing, troubleshooting and performance optimisation.
• Lead technical due diligence and the integration of software products, ensuring solutions meet operational, architectural and security requirements.
• Collaborate with operations teams to ensure applications are deployment ready, maintainable and supported effectively in live environments.
• Work alongside security architects and accreditation teams to ensure data solutions comply with relevant security, regulatory and legal requirements.
• Advise clients, project managers and technical stakeholders on technical approaches, delivery estimates and the implications of user stories and requirements.
• Contribute to technical proposals, solution design and pre sales activities, supporting the development of new client opportunities.
• Lead, manage and develop a small team of engineers, providing technical direction, performance management, mentoring and career development support.

Essential Skills and Experience

• Proven experience leading software engineering teams delivering complex, data intensive systems and components.
• Strong commercial software development experience using at least one of Java, Scala or Python.
• Extensive commercial experience working with Palantir, with the ability to apply the platform within enterprise data engineering solutions.
• Practical experience developing data processing solutions using cloud platforms and technologies such as AWS, Microsoft Azure, Google Cloud Platform or Databricks.
• Expert level SQL skills, including the use of SQL extensions for analytical data processing and engineering.
• Strong understanding of distributed data stores, distributed computing and large scale data processing frameworks.
• Experience establishing and applying software design patterns, coding standards, automated testing and deployment practices.
• Proficient in designing and developing analytical and operational data models.
• Ability to communicate complex technical designs and architectural concepts clearly to both technical and non technical stakeholders.
• Knowledge of data warehousing methodologies, architectures and techniques.
• Relevant AWS, Azure or Google Cloud certifications, particularly within data engineering and data services.
• A passion for continuous improvement, actively contributing to engineering standards and promoting data engineering best practice.
• Experience sharing technical knowledge, contributing to development communities or participating in technology forums.