Associate Consultant - Data Engineering

Optima Partners; City of Edinburgh; Hybrid; GBP 28,000 - 44,000; Full time

Benefits offered by this job

Private medical insurance

Salary sacrifice pension scheme

Up to 37 days holiday

Discretionary bonus plan (up to 15%)

Access to perks portal

Dedicated development time and funding

Job summary

Optima Partners, an Edinburgh-based data and business consultancy, seeks an Associate Consultant - Data Engineering to design, build and maintain scalable data pipelines for a diverse client base. You will work in cross-functional teams with data scientists, engineers and consultants while developing practical data solutions.

The role offers exposure to modern data engineering practices, with mentoring and opportunities for career progression within a collaborative, knowledge-sharing culture.

Qualifications

• Working knowledge of Python and SQL for data processing.
• Exposure to Spark, Databricks or similar big data technologies.
• Awareness of Databricks, Delta Lake, notebooks or jobs; production experience not essential.
• Familiarity with AWS, Azure or GCP for data tasks.
• Basic understanding of relational databases and SQL queries.
• Familiarity with Git and collaborative workflows.
• Interest in CI/CD, automated testing and infrastructure as code.

Responsibilities

• Support the development, testing and maintenance of data pipelines that collect, transform and integrate data from a range of sources.
• Assist in implementing and improving data architectures, including data warehouses and data lakes.
• Help build and maintain ETL and ELT workflows with exposure to Databricks Jobs and Delta Lake patterns.
• Work with data scientists, analysts, engineers and client stakeholders to understand requirements and deliver data solutions.
• Support monitoring, troubleshooting and optimisation of data systems under senior guidance.
• Contribute to automation opportunities and improve data engineering workflows.
• Maintain clear technical documentation for data solutions and learning materials.

Skills

Python

SQL

Spark

Databricks

AWS

Azure

GCP

PostgreSQL

Git

CI/CD

Agile

Tools

Databricks

Delta Lake

Job description

Who we are

Optima Partners is an advanced data and business consultancy headquartered in Edinburgh, UK. We are a practitioner-led organisation that collaborates with top consumer brands to drive transformation and foster customer-centricity through our expertise in customer strategy, innovative design, and advanced data science and engineering.

We work with leading consumer brands to tackle and overcome complex business and customer problems, drive transformation and champion customer-centric agendas. We are proud to include leading UK and global brands among our current clients, including Lloyds Banking Group, NatWest Group, Bank of Ireland, Nationwide, Aviva, Biogen, Eon Next, OVO, Virgin Media O2, BT, HMD Global, Centrica and GSK. We are obsessive about delivering value for our clients and work in a collaborative, engaged and creative way with our colleagues and clients. We strive to support the transition of knowledge and capability into strategic teams.

The opportunity

We are looking for an Associate Consultant - Data Engineering to support the design, build, testing and maintenance of data pipelines, workflows and data solutions for our diverse client base. Working within dynamic, multi-disciplinary project teams, you will learn from experienced engineers while contributing to scalable, reliable data solutions that follow good engineering practices.

This is an exciting opportunity for an enthusiastic data engineering professional with up to one year of commercial experience, or equivalent project experience, to develop consulting and technical skills across a variety of commercial sectors. At Optima, you will work in cross-functional teams, collaborating with experts across data science, engineering, architecture, product and consulting while gaining exposure to varied business challenges.

Optima provides an inclusive space for professional development, with a strong focus on continuous learning, mentoring and knowledge sharing. With the chance to work alongside experienced engineers, scientists, data architects and product specialists, you will build practical skills in modern data engineering while contributing to real client projects and developing a clear path for career progression.

What you will be doing

• Support the development, testing and maintenance of data pipelines that collect, transform and integrate data from a range of sources.

• Assist in implementing and improving data architectures, including data warehouses, data lakes and other storage solutions under the guidance of senior team members.

• Help build and maintain ETL and ELT workflows that support data quality and accessibility, including exposure to Databricks Jobs, Workflows and Delta Lake patterns where appropriate.

• Work closely with data scientists, analysts, engineers and client stakeholders to understand requirements, ask good questions and support the delivery of practical data solutions.

• Support monitoring, troubleshooting and optimisation activities for data systems, learning how to improve performance, reliability and cost efficiency with guidance from senior colleagues.

• Contribute to automation opportunities and learn how modern tools can improve data engineering workflows.

• Maintain clear technical documentation for data solutions, processes and learning materials to support project delivery and knowledge sharing.

What skills we would like you to have

• Working knowledge of Python, SQL or similar languages for data processing, with a willingness to develop deeper engineering capability.

• Some exposure to Spark, Databricks or similar big data technologies through commercial work, internships, training, personal projects or academic experience.

• Awareness of Databricks, Delta Lake, notebooks, jobs or workflows is beneficial; hands-on production experience is not essential and can be developed in role.

• Familiarity with AWS, Azure, GCP or other cloud ecosystems for data engineering tasks is desirable.

• Basic understanding of relational databases such as PostgreSQL or SQL Server, including writing and interpreting SQL queries.

• Familiarity with Git and collaborative workflows such as branching, pull requests and code review.

• An interest in how code gets from a pull request into production, including CI/CD pipelines, automated testing and infrastructure as code such as Terraform.

• Up to one year of commercial experience in data engineering, consulting or a related technical role; internships, placements, graduate schemes, bootcamps or strong project experience will also be considered.

• Some exposure to working in a team environment, client-facing environment or on collaborative technical projects is desirable.

• Awareness of Agile or Scrum ways of working is beneficial but not essential.

What you will bring

• Strong problem-solving and analytical thinking abilities.

• Good communication skills, with the ability to ask clear questions, share progress and explain technical ideas in an accessible way.

• A team-oriented mindset with curiosity, adaptability and a willingness to learn from feedback.

This is an exciting opportunity to work directly on client projects, building practical data engineering capability while learning from experienced colleagues in a specialist data and AI consultancy.

What we offer

• Competitive base salary.

• Inclusion in our annual discretionary bonus plan with an on-target performance bonus of up to 15%.

• Up to 37 days holiday inclusive of personal and public allocations. Seven are fixed days for Christmas and New Year, and 30 are floating days taken at your discretion, subject to client scheduling and line manager approval.

• Private medical insurance, single cover.

• Group life and income protection insurance.

• Salary sacrifice pension scheme after three months' employment.

• Access to over 1,000 perks and discounts via an employee discount portal.

• Dedicated development time plus tools and funding to support personal and professional development.