Senior/Lead Data Engineer

Cognizant; Greater London; On-site; GBP 40,000 - 60,000; Full time

Job summary

An established industry player seeks a skilled data engineer to design and deliver Azure-based data platform solutions. This role involves writing Python and Spark SQL code, implementing data validation rules, and managing data integration test cases. The ideal candidate will have strong experience with Azure technologies, including ADF and Databricks, and a solid understanding of data modeling. You'll work in a dynamic environment, contributing to innovative projects while ensuring high standards of data quality and performance. If you're passionate about data engineering and eager to make a significant impact, this opportunity is perfect for you.

Qualifications

• Strong experience in Azure-based data platform solutions.
• Hands-on experience in Python and knowledge of others.
• Knowledge in data modeling and agile delivery.

Responsibilities

• Write Python/PySpark and Spark SQL code for data validation.
• Implement data validation and curation rules.
• Create data integration test cases and manage pipelines.

Skills

Python

PySpark

Spark SQL

Azure ADF

Databricks

Kafka

Data modeling

Agile delivery

C#

Tools

Azure DevOps

GitHub

Power BI

Tableau

Job description

Please note that this role requires security clearance at SC level. You must be SC cleared to be considered for the role.

Tasks and Responsibilities:

Engineering:

• Ingestion configuration.
• Write Python/PySpark and Spark SQL code for validation/curation in notebook.
• Create data integration test cases.
• Implement or amend worker pipelines.
• Implement data validation/curation rules.
• Convert data model into technical data mapping.
• Implement integrations.
• Data migration implementation and execution.
• Data analysis (profiling, etc.).
• Azure DevOps/GitHub configurations for ADF and Databricks code.

Our Ideal Candidate:

Strong experience in designing and delivering Azure-based data platform solutions, including:

1. Azure ADF and Databricks, Azure Functions, App Service, Logic Apps, AKS, Azure App Service, Web App.
2. Good knowledge of real-time streaming applications, preferably with experience in Kafka real-time messaging or Azure Functions, Azure Service Bus.
3. Spark processing and performance tuning.
4. File formats partitioning (e.g., Parquet, JSON, XML, CSV).
5. Azure DevOps/GitHub.

• Hands-on experience in at least one of Python with knowledge of the others.
• Experience with synchronous and asynchronous interface approaches.
• Knowledge in data modeling (3NF/Dimensional modeling/Data Vault 2).
• Work experience in agile delivery.
• Able to provide comprehensive documentation.
• Able to set and manage realistic expectations for timescales, costs, benefits, and measures for success.

Nice to Have:

• Experience with integration and implementation of data cataloging tools like Azure Purview/Collabra.
• Experience in implementing and integrating visualization tools like Power BI/Tableau, etc.
• Experience in C# application development (ASP.NET MVP).