Pyspark Developer - Financial Services

Vallum; Greater London; On-site; GBP 50,000 - 80,000; Full time

Job summary

A UK-based technology firm is seeking a skilled professional experienced in programming under the Microsoft Azure Cloud Platform. The ideal candidate should have a strong background in Spark Programming, data processing, and building ETL workflows. Proficiency in Python, Git, and Azure Data Lake environments is essential, along with effective communication skills for stakeholder management. This position offers an opportunity to work in a dynamic team and contribute to innovative data processing solutions.

Qualifications

• Experience of programming under Microsoft Azure Cloud Platform.
• Knowledge of Spark programming for large scale data processing.
• Hands-on experience with lakehouses and dataflows.

Skills

Microsoft Azure Cloud Platform

Spark Programming

ETL workflows

Git

Python

PySpark

Java

Database knowledge

Tools

Gitlab

CICD pipeline

Job description

Essential skills/knowledge/experience:

• Experience of programming under Microsoft Azure Cloud Platform
• Have knowledge of Spark Programming Ability to write Spark code for large scale data processing, including RDDs, DataFrames, and Spark SQL
• Hands‑on experience with lakehouses, dataflows, pipelines, and semantic models
• Ability to build ETL workflows
• Familiarity with time‑series data, market feeds, transactional records, and risk metrics
• Familiarity with Git, DevOps pipelines, and automated deployment
• Strong communication skills with a collaborative mindset to work with and manage stakeholders
• Experience of programming under Microsoft Fabric Platform
• Python/Notebook programming
• PySpark programming
• Spark Streaming/batch processing
• Delta table Optimization
• Fabric Spark jobs
• Java programming language, OOP knowledge
• Database knowledge, including Relational Database and Non‑SQL database.
• Experience of using the tools: Gitlab, Python unit test, CICD pipeline.
• Good skill of troubleshooting
• Familar with the Agile. Well communication.
• Good English listening and speaking for communicating requirements and development tasks/issues

Desirable skills/knowledge/experience:

• Ability to prepare and process datasets for Power BI usage
• Experience with OneLake, Azure Data Lake, and distributed computing environments
• Understanding of financial regulations such as GDPR, SOX etc.
• Spark application performance tuning
• Knowledge of Docker/Kubernetes