Senior Data Engineer

Social network you want to login/join with:

Senior Data Engineer opportunity to join a well-known financial services company working on a Remote first basis (once a month in the office on average).

Senior Data Engineer - Remote First (offices in Lake District) - Up to £65,000 + Great Benefits

Purpose of Job

Advise on and ensure the maximum return of value from the company's data assets through the use of best practice data engineering techniques that aid the delivery of the company's data management strategy and roadmap, aligning with regulatory requirements.

Contribute towards the support, maintenance, and improvement of the company's platform, including data pipelines, ensuring data quality and governance.

Responsibilities

1. Advise on the design, implementation, and maintenance of complex data engineering solutions.
2. Ensure the implementation and maintenance of data solutions to acquire and prepare data, adhering to best practices (Extract, Transform, Load).
3. Contribute to and ensure the delivery of data pipelines that connect data within and between data stores, applications, and organizations.
4. Ensure the delivery of complex data quality checks and remediation.
5. Identify data sources, processing concepts, and methods.
6. Design and implement on-premise, cloud-based, and hybrid data engineering solutions.
7. Structure and store data for uses including analytics, machine learning, data mining, and sharing with applications and organizations.
8. Harvest structured and unstructured data.
9. Integrate, consolidate, and cleanse data.
10. Migrate and convert data.
11. Apply ethical principles and regulatory requirements in handling data.
12. Ensure appropriate storage of data in line with legislation and company requirements.
13. Guide and contribute to the development of junior and trainee Data Engineers.
14. Provide technical guidance to Data Engineers.

Knowledge, Experience, and Qualifications

1. Excellent knowledge and experience in Azure Data services, including DataLake, Data Factory, DataBricks, Azure SQL (Indicative experience = 5+ years).
2. Build and test processes supporting data extraction, transformation, data structures, metadata, dependency, and workload management.
3. Knowledge of Spark architecture and modern Data Warehouse/Data Lake/Lakehouse techniques.
4. Build transformation tables using SQL.
5. Moderate knowledge of Python/PySpark or equivalent programming languages.
6. Experience with PowerBI Data Gateways, DataFlows, and permissions.
7. Creation, optimization, and maintenance of relational SQL and NoSQL databases.
8. Experience working with CI/CD tools such as Azure DevOps or GitHub, including repositories, source control, pipelines, and actions.
9. Awareness of Informatica or similar data governance tools (desirable).
10. Experience working in agile (SCRUM) and waterfall delivery teams.
11. Experience with Confluence and Jira.
12. Experience in a financial services or other highly regulated environment.
13. Interest or experience in AI is desirable.