Data Engineer

GIS Recruiter Ltd; Greater London; On-site; GBP 65,000 - 90,000; Full time

Job summary

GIS Recruiter Ltd seeks an experienced Data Engineer to support a data migration and integration programme in a relatively low maturity Azure environment. You will design, build and maintain data pipelines, migrating data from diverse legacy systems to Azure Data Lake and Databricks.

You will also integrate data into a proprietary transformation solution and work with a range of source systems including Oracle, Oracle Fusion, Informatica and SharePoint, contributing to the maturity of the Azure

Qualifications

• Experience designing and implementing data pipelines in Azure cloud environments.
• Proven ability to migrate data from diverse legacy and enterprise sources.

Responsibilities

• Design, build and maintain data migration and ingestion pipelines within Azure.
• Build integrations between source systems, Azure Data Lake and Databricks.
• Develop and manage pipelines using Azure Data Factory (ADF).
• Extract and migrate data from Oracle, Oracle Fusion, Informatica, SharePoint and other sources.
• Help define scalable data ingestion, transformation and migration patterns for future source systems.
• Support data quality, validation and troubleshooting throughout the migration.

Skills

Azure Data Engineering

Azure Data Factory

Azure Data Lake

Databricks

ETL Pipelines

Large-scale migration

Oracle Fusion

SharePoint Integration

Source System Integration

Data Transformation

Tools

Azure Data Factory (ADF)

Azure Data Lake

Databricks

Job description

We are looking for an experienced Data Engineer to support a data migration and integration programme within a relatively low maturity Azure environment.

The role will focus on designing and building robust data pipelines, migrating data from a wide range of legacy and enterprise source systems, and establishing scalable data flows into Azure Data Lake and Databricks. A key part of the role will also involve integrating data into a proprietary data transformation solution where applicable.

Key Skills & Experience

• Azure Data Engineering
• Azure Data Factory (ADF)
• Azure Data Lake
• Databricks
• Building ETL / ELT pipelines
• Large-scale data migration and integration
• Oracle and/or Oracle Fusion data sources
• SharePoint data extraction and integration
• Working with multiple and varied source systems
• Data transformation, mapping and validation
• Designing data pipelines in environments where architecture and processes are still developing

Key Responsibilities

• Design, build and maintain data migration and ingestion pipelines within Azure.
• Support the setup and development of the Azure Data Lake environment.
• Build integrations between source systems, Azure Data Lake and Databricks.
• Develop and manage pipelines using Azure Data Factory (ADF).
• Extract and migrate data from a variety of source systems, including:
  • Oracle
  • Oracle Fusion
  • Informatica
  • SharePoint, which will be an initial priority
  • Additional enterprise and legacy systems as the programme develops
• Help define scalable data ingestion, transformation and migration patterns for future source systems.
• Support data quality, validation and troubleshooting throughout the migration process.
• Work closely with technical and business stakeholders to understand source data, target structures and transformation requirements.
• Help improve the overall maturity of the Azure data environment, introducing appropriate engineering standards and best practices.