Lead Data Engineer

FXC Intelligence; Camden Town; Hybrid; GBP 70,000 - 90,000; Full time

Job summary

A leading data intelligence provider in Camden Town is seeking a Lead Data Engineer to manage the Data Ingest team. The role involves overseeing data scraping processes, enhancing data quality, and leading migration efforts to AWS. Candidates should possess advanced Python and SQL skills, along with experience in web scraping and team management. This position offers a hybrid working environment and attractive benefits including a substantial holiday allowance and a learning budget.

Benefits

25 days holiday

Health coverage

Annual bonus

Pension scheme

Enhanced maternity/paternity policy

Learning budget

Quarterly coaching sessions

Qualifications

• Advanced skills in Python and SQL.
• Experience in web scraping and/or backend development.
• A proactive approach with the ability to set and exemplify high standards.

Responsibilities

• Lead and manage the Data Ingest team, providing vision and guidance.
• Translate strategic roadmap into actionable tasks.
• Implement new scrapers and maintain existing ones.

Skills

Python

SQL

Web scraping

Backend development

AI for productivity

Excellent communication

Tools

PostgreSQL

RabbitMQ

Redis

Puppeteer

Selenium

Pandas

Numpy

Scikit-learn

Job description

We are looking for a Lead Data Engineer to lead our Data Ingest team, responsible for scraping the data that fuels our products. You'll play a pivotal role in shaping how we capture, process and scale the data pipelines that underpin our insights.

Responsibilities

• Lead and manage the Data Ingest team, providing vision and guidance.
• Translate strategic roadmap into actionable tasks and deliver progress visibility to stakeholders.
• Conduct or oversee research and technical investigations, prioritising improvements based on stakeholder priorities.
• Improve and automate team processes, including releases, cross‑team collaboration, and fostering a culture of innovation and growth.
• Implement new scrapers and maintain existing ones.
• Lead migration of scrapers to AWS and transition from old Jenkins‑based orchestration to a new micro‑service‑based system.
• Enhance the scraping platform in data quality, performance, and observability.

About the Team

Join a technical team of around 40 technologists dedicated to continuously and reliably collecting large amounts of financial data describing cross‑border payments, checking quality, and serving it to our users. The Data Ingest team is the oldest at FXC Intelligence, with over six years of history and a platform currently being migrated to its fourth generation. We maintain over 100 scrapers powering most of the company's critical datasets.

Qualifications

• Advanced skills in Python and SQL.
• Experience in web scraping and/or backend development.
• Experience utilising AI to increase team productivity, efficiency, and data quality.
• A proactive approach with the ability to set and exemplify high standards.
• Ability to work in a fast‑paced environment and adapt to change quickly.
• Excellent communication skills.
• Proven experience and a desire to manage a team.
• A desire to understand and engage with the business domain and product.
• A love for learning and improving your skills.

Preferred (but not essential)

• Prior experience working for a data product company or in a big‑data role.
• Experience with advanced database management and data modelling.
• Experience building or maintaining third‑party or in‑house data quality and cataloguing solutions.
• Experience with DBT, Airflow, Kafka/RabbitMQ.
• Experience documenting system architecture.
• Experience collaborating with BI, DA and/or ML teams.

Tech Stack

• Python, PostgreSQL, RabbitMQ, Redis, Puppeteer, Playwright, Selenium, Pandas, Numpy, scikit‑learn.

Company Overview

FXC Intelligence is a leading provider of cross‑border payments data and intelligence, serving some of the world's biggest companies, central banks and NGOs with strategic insights, expertise and awareness to compete in their chosen markets. Joining the team means diving into data‑driven exploration and innovation, revolutionising financial insights through cutting‑edge technologies, machine learning and predictive analytics.

Benefits

• 25 days holiday per annum (plus bank holidays), increasing to 26 days after 2 years of service.
• Health coverage for you and your family with AXA.
• Annual bonus / option scheme.
• Pension scheme.
• Enhanced maternity and paternity policy.
• After four years of employment, you are entitled to a fully paid 6‑week sabbatical.
• After one year of employment, you can work abroad for up to three months of the year with manager approval.
• Quarterly external one‑to‑one coaching sessions.
• £1,000 annual learning budget for courses, books and conferences.
• Annual salary reviews with your manager.
• Opportunity to work with a vibrant, friendly team in a relaxed but committed culture.
• Monthly London‑based socials paid for by the company.
• Hybrid working environment, with expectation to visit the London Liverpool Street office each month.
• Access to LGBTQIA+ and Women in Fintech employee networks.
• Recognised as one of The Sunday Times Best Places to Work in 2025.

Inclusion Statement

FXC Intelligence is committed to creating an inclusive culture that welcomes diverse talent, and all applicants will be considered without regard to ethnicity, religion, sexual orientation, gender identity, family or parental status, national origin, veteran status, neurodiversity status or disability status.