Senior Cloud Platform Data Engineer / Cloud Architect

Avance Consulting; London; Hybrid; GBP 60,000 - 100,000; Full time

Job summary

An innovative company is seeking a Senior Cloud Platform Data Engineer to join their Payment Gateway team. This hybrid role involves designing and implementing data solutions on cloud platforms, primarily AWS, ensuring high standards of quality and performance. You will be a subject matter expert, guiding the team in best practices and technical leadership. Collaborate with various stakeholders to translate data needs into effective solutions while maintaining data quality and governance. If you are passionate about cloud technologies and have a strong background in data engineering, this is the perfect opportunity to make a significant impact in a dynamic environment.

Qualifications

• 7+ years in data engineering with a focus on cloud platforms, especially AWS.
• Experience in CI/CD pipelines and strong analytical skills.

Responsibilities

• Design and implement complex data solutions on cloud platforms.
• Drive the development of data pipelines and ETL/ELT processes.

Skills

Cloud Data Engineering

Data Pipeline Development

Data Quality Management

Analytical Skills

Communication Skills

Problem-Solving Skills

DevOps Methodologies

Agile Development

Education

Bachelor's Degree in Computer Science or related field

Tools

AWS Glue

AWS Lambda

AWS S3

AWS Redshift

AWS EMR

Jenkins

Docker

Kubernetes

Kafka

Kinesis

Job description

We are seeking a highly skilled and experienced Senior Cloud Platform Data Engineer (SME) to join our dynamic Payment Gateway team in Fleet UK for a Full time Hybrid role. As an SME you will serve as a subject matter expert providing technical leadership guidance and best practices across the team.

You will play a critical role in designing, developing, and implementing complex data solutions on cloud platforms (primarily AWS) ensuring they meet the highest standards of quality, scalability, and performance.

1. Serve as a subject matter expert in cloud data engineering providing technical guidance and mentorship to the team.
2. Drive the design, development, and implementation of complex data pipelines and ETL/ELT processes using cloud-native technologies (e.g. AWS Glue, AWS Lambda, AWS S3, AWS Redshift, AWS EMR).
3. Develop and maintain data quality checks, data validation rules, and data lineage documentation.
4. Collaborate with data analysts, data scientists, business stakeholders, and product owners to understand their data needs and translate them into innovative and effective technical solutions.
5. Troubleshoot and resolve complex data quality issues and performance bottlenecks.
6. Participate in the evaluation, selection, and implementation of new data technologies and tools.
7. Adhere to and champion best practices in data engineering including data security, privacy, and governance.
8. Define and enforce data engineering standards, processes, and best practices within the team.
9. Develop and maintain CI/CD pipelines for data engineering workflows using tools like Jenkins.
10. Automate data ingestion, transformation, and loading processes.
11. Work closely with DevOps teams and other engineering teams to ensure smooth deployment and operations of data pipelines.
12. Experience working on Linux platforms and managing data on Linux servers.
13. Stay abreast of the latest advancements in cloud computing and data engineering technologies.
14. Contribute to the development and improvement of the team's overall performance and efficiency.
15. May lead small projects or initiatives within the data engineering team.

Your Profile

1. 7+ years of experience in data engineering with a strong focus on cloud platforms (preferably AWS).
2. 3+ years of experience as a Senior Engineer or SME in a data engineering role.
3. Solid experience working on Linux platforms. Experience with CI/CD pipelines and tools like Jenkins.
4. Strong understanding of DevOps methodologies and principles.
5. Solid understanding of data warehousing, data modeling, and data integration principles.
6. Proficiency in at least one scripting/programming language (e.g. Python, Scala, Java). Experience with SQL and NoSQL databases.
7. Familiarity with data quality and data governance best practices.
8. Strong analytical and problem-solving skills.
9. Excellent communication, interpersonal, and presentation skills.

Desired Skills:

1. Experience with containerization technologies (e.g. Docker, Kubernetes).
2. Experience with data streaming platforms (e.g. Kafka, Kinesis).
3. Experience with data visualization and business intelligence tools.
4. Experience with Agile development methodologies.
5. AWS Certifications (e.g. AWS Certified Data Analytics - Specialty).