Data Engineer

N Consulting Ltd; London; Hybrid; GBP 100,000 - 125,000; Full time

Job summary

This innovative firm is seeking a skilled Data Engineer to design and optimize ETL processes, ensuring data integrity across various platforms. In this hybrid role, you will collaborate with cross-functional teams to build scalable data pipelines and enhance existing architectures. Your expertise in SQL Server and Snowflake will be crucial for transforming and managing large datasets effectively. Join a dynamic environment where your contributions will drive impactful data solutions and support strategic business initiatives. If you are passionate about data engineering and eager to tackle complex challenges, this opportunity is perfect for you.

Qualifications

• 10-12 years of experience in data engineering and ETL processes.
• Strong proficiency in SQL and Snowflake architecture.

Responsibilities

• Design and maintain ETL processes for data movement to Snowflake and SQL Server.
• Collaborate with stakeholders to define data strategies and ensure data quality.

Skills

SQL Server

Snowflake

ETL Processes

AWS

Data Warehousing

Data Modeling

GitLab

Tools

Informatica PowerCenter

AutoSys

Airflow

SQL Server Agent

Job description

Role : Data Engineer
Work Mode : Hybrid
Contract Role
Location : London OR Edinburgh
Experience : 10 -12 Years

Job Description:

1. Design, develop, and maintain ETL processes to move data from source systems to Snowflake, Teradata and SQL Server environments.
2. Build scalable data pipelines for processing and storing large datasets.
3. Collaborate with analysts and business stakeholders to define and implement data strategies.
4. Optimize database performance, including query optimization, indexing, and partitioning across all enterprise data platforms.
5. Work with cross-functional teams to gather requirements and develop effective data solutions.
6. Ensure data quality, consistency, and integrity throughout the data lifecycle.
7. Maintain and improve existing data architectures and workflows to meet business requirements.
8. Create and maintain documentation for data systems, pipelines, and processes.
9. Monitor and troubleshoot data pipeline performance issues and resolve them promptly.
10. Assist in the migration and integration of data from legacy systems into Snowflake and other strategic data stores.
11. Perform data transformation tasks using SQL, Snowflake SQL, and relevant data processing tools.

Required Skills and Qualifications:

1. Proven experience working with SQL Server (e.g., T-SQL, Stored Procedures, Indexing, Query Optimization, System Catalog Views).
2. Strong experience in Snowflake architecture, including data loading, transformation, and performance tuning.
3. Proficient in ETL processes using tools such as Informatica PowerCenter and BDM, AutoSys, Airflow, and SQL Server Agent.
4. Experience with cloud platforms, preferably AWS.
5. Strong knowledge of AWS cloud services, including EMR, RDS Postgres, Redshift Athena, S3, and IAM.
6. Solid understanding of data warehousing principles and best practices.
7. Strong proficiency in SQL for data manipulation, reporting, and optimization.
8. Knowledge of data modeling and schema design.
9. Experience working with large, complex datasets and implementing scalable data pipelines.
10. Familiarity with version control tools such as GitLab.
11. Experience with data integration, data governance, and security best practices.