AWS Data Engineer

Datatech Analytics; London; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

An established industry player in consumer behavior analytics is seeking a motivated AWS Data Engineer to enhance their data infrastructure. In this dynamic role, you will collaborate with a talented team to develop advanced data pipelines and integrate innovative technologies like DataOps and Generative AI. Your expertise in AWS services, SQL, and Python will be crucial in optimizing ETL processes and ensuring data security. This is an exciting opportunity to challenge yourself in a collaborative environment, working with diverse data and cutting-edge technologies. If you're ready to make a significant impact, this role is for you!

Qualifications

• Proven experience with AWS Cloud platforms and data pipeline orchestration.
• Strong SQL and Python skills for data transformation and cleaning.

Responsibilities

• Develop and optimize ETL/ELT processes for data integrity.
• Implement CI/CD pipelines to automate data workflows.

Skills

AWS services

SQL

Python

Pandas

Dagster

DataOps

CI/CD

Data transformation

Communication skills

Education

Bachelor's degree in Computer Science or related field

Tools

AWS Lambda

AWS Glue

AWS Athena

AWS RDS

AWS S3

Snowflake

Airflow

Job description

AWS Data Engineer
Salary: Negotiable to £80,000 Dependent on Experience
Location: London (Hybrid working 3 days per week in the office, 2 days home-based)
Job Ref: J12931

A leader in consumer behaviour analytics seeks a driven AWS Data Engineer to guide data infrastructure architecture, working alongside a small talented team of engineers, analysts, and data scientists. In this role, you'll enhance the data platform, develop advanced data pipelines, and integrate cutting-edge technologies like DataOps and Generative AI, including Large Language Models (LLMs).

This is an exciting opportunity for someone looking to challenge themselves in a collaborative environment, working with a breadth of diverse data and cutting-edge technologies. You'll have proven experience developing AWS Cloud platforms end to end, orchestrating data using Dagster or similar, as well as coding in Python and SQL.

Key Responsibilities

1. Develop and optimize ETL/ELT processes to support data transformation and integrity for analytics.
2. Explore and evaluate new data warehousing solutions, including Snowflake, to improve data accessibility and scalability.
3. Partner with product and engineering teams to define data architecture and best practices for reporting.
4. Ensure data security, compliance, and governance across data systems.
5. Implement and maintain CI/CD pipelines to automate data workflows and enhance system reliability.
6. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability and performance.

Essential Skills and Experience:

1. Hands-on experience with AWS services, including Lambda, Glue, Athena, RDS, and S3.
2. Strong SQL skills for data transformation, cleaning, and loading.
3. Strong coding experience with Python and Pandas.
4. Experience with any flavour of data pipeline and workflow management tools: Dagster, Celery, Airflow, etc.
5. Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
6. Experience supporting and working with cross-functional teams in a dynamic environment.
7. Strong communication skills to collaborate with remote teams (US, Canada).

Nice to Have

1. Familiarity with LLMs including fine-tuning and RAG.
2. Knowledge of Statistics.
3. Knowledge of DataOps best practices, including CI/CD for data workflows.

Additional Requirements

Candidates must have an existing and future right to live and work in the UK. Sponsorship at any point is not available.

If this sounds like the role for you, then please apply today!

Alternatively, you can refer a friend or colleague by taking part in our fantastic referral schemes! If you have a friend or colleague who would be interested in this role, please refer them to us. For each relevant candidate that you introduce to us (there is no limit) and we place, you will be entitled to our general gift/voucher scheme.

Datatech is one of the UK's leading recruitment agencies in the field of analytics and host of the critically acclaimed event, Women in Data. For more information, visit our website: www.datatech.org.uk