Data Engineer

Pyramid Consulting, Inc; Greater London; Remote; GBP 90,000 - 130,000; Full time

Job summary

Pyramid Consulting, Inc. is seeking a Senior Data Engineer in a remote-capable role covering London, Spain, Germany, and Portugal.

The candidate will own end-to-end data engineering and AI workstreams, delivering scalable solutions using Python, SQL, Spark, and modern MLOps practices. Responsibilities include leading technical initiatives, mentoring engineers, and building Generative AI/Agentic AI capabilities with robust governance, security, and production-readiness across AWS, Azure, and GCP

Qualifications

• Degree in Computer Science, Engineering or equivalent.
• 5+ years of relevant Data Engineering experience.
• Expert-level Python and SQL experience.
• Experience with Generative AI, Agentic AI, ML, and BI solutions.
• Knowledge of RAG, embeddings, vector databases, and prompt engineering.

Responsibilities

• Design, develop, and maintain scalable data engineering and AI solutions.
• Lead technical workstreams and mentor junior engineers.
• Build Generative AI, Agentic AI, ML, and BI solutions.
• Work with Spark, LangChain, Databricks, Airflow, Dagster, Dask, Kedro.
• Develop AI capabilities including prompt engineering, embeddings, and context construction.
• Own AI feature delivery from development to production with guardrails and performance optimisation.
• Implement data security, governance, authentication, and RBAC.
• Establish MLOps/LLMOps pipelines with CI/CD and AI evaluation.

Skills

Python

SQL

Generative AI

Agentic AI

MLOps

LLMOps

CI/CD

RBAC

Data governance

Education

Degree in Computer Science, Engineering

Tools

Spark

Databricks

LangChain

Airflow

Dagster

Dask

Kedro

Terraform

Job description

Senior Data Engineer – London-UK, Spain, Germany & Portugal

Remote

Role Overview

We are looking for a Senior Data Engineer with strong expertise in Python, SQL, Data Engineering, and AI/ML to design, build, and deliver scalable data and AI solutions.

The role involves owning technical workstreams end-to-end, contributing to Generative AI and Agentic AI solutions, and applying modern data engineering, MLOps, and LLMOps practices across enterprise environments.

Key Responsibilities

• Design, develop, and maintain scalable data engineering and AI solutions.
• Lead technical workstreams and mentor junior engineers while promoting strong software engineering practices.
• Build Generative AI, Agentic AI, Machine Learning, and Business Intelligence solutions.
• Work with technologies such as Spark, LangChain, Databricks, Dask, Airflow, Dagster, and Kedro.
• Develop AI capabilities using prompt engineering, RAG, embeddings, vector databases, and context engineering.
• Own AI feature delivery from development through production, including evaluation, reliability, guardrails, and performance optimisation.
• Implement data security and governance, including PII/PHI protection, authentication, and RBAC.
• Establish and maintain MLOps and LLMOps pipelines, including CI/CD and automated AI/agent evaluation.
• Use tools such as LangSmith, Opik, Langfuse, and Terraform across AWS, Azure, and GCP environments.
• Work effectively in polyglot enterprise environments involving Python, SQL, Scala, and Java.
• Independently manage technical deliverables, priorities, and timelines.
• Collaborate effectively with technical and business stakeholders.
• Travel occasionally when required.

Required Skills & Experience

• Degree in Computer Science, Engineering, or equivalent professional experience.
• 5+ years of relevant experience in Data Engineering.
• Strong experience leading technical initiatives, mentoring engineers, and driving engineering best practices.
• Expert-level Python and SQL skills.
• Hands-on experience with Generative AI, Agentic AI, Machine Learning, and BI solutions.
• Strong knowledge of RAG, embeddings, vector databases, prompt engineering, and context construction.
• Experience with modern frameworks and platforms such as Spark, Databricks, LangChain, Airflow, Dagster, Dask, or Kedro.
• Experience with MLOps/LLMOps, CI/CD, AI evaluation, and production AI systems.
• Knowledge of LangSmith, Opik, Langfuse, or similar AI evaluation/observability tools.
• Experience with Terraform and cloud environments such as AWS, Azure, and GCP.
• Understanding of data security, governance, PII/PHI, authentication, and RBAC.
• Strong English communication skills and proficiency in the relevant local language.
• Ability to work independently with strong ownership, organisation, and time-management skills.

Nice to Have

• Experience using AI coding assistants/agents such as Cursor, Claude Code, or Codex.
• Experience working with Scala and Java in enterprise environments.
• Willingness to travel when required.

Location: Spain, Germany or Portugal, UK