Data Engineer: AI Pipelines & Insight Dashboards

Optima Partners; City of Edinburgh; On-site; GBP 42,000 - 54,000; Full time

Benefits offered by this job

Competitive salary

Annual bonus 15%

37 days holiday

Private medical insurance

Life and income protection

Pension scheme

Employee discounts

Professional development funding

Job summary

Optima Partners in Edinburgh is seeking a Data Engineer to help design, build and maintain reliable data pipelines for clients across diverse sectors. This early-career role suits someone with up to two years commercial experience or equivalent project work who wants to develop technical and consulting skills.

You'll work alongside data engineers, data scientists and product specialists, with mentoring and hands-on training to grow in modern data platforms, dashboards and data storytelling,

Qualifications

• Working knowledge of Python, SQL or similar data processing languages.
• Some exposure to Spark or Databricks through work, study, training or projects.
• Awareness of Delta Lake, notebooks and workflows; production experience is not essential.
• Familiarity with cloud platforms such as AWS, Azure or GCP is desirable.
• Basic understanding of relational databases and SQL queries.
• Awareness of tools such as Airflow, Apache NiFi or Talend is helpful.
• Foundations in data warehousing, modelling and schema design, with an interest in building dashboards using Power BI, QlikView and Looker to analyse and visualise data and communicate insights to clients.
• Familiarity with Git and an interest in automated testing, CI/CD and collaborative development.
• Up to one year's relevant commercial experience, or experience through internships, placements, graduate schemes, bootcamps or strong projects.
• Team, client-facing or collaborative project experience is desirable.
• Awareness of Agile or Scrum is beneficial

Responsibilities

• Help build, test and maintain data pipelines using tools such as Databricks.
• Support data warehouse and data lake development with guidance from experienced engineers.
• Develop ETL/ELT workflows that make reliable, high-quality data accessible.
• Collaborate with colleagues and clients to understand requirements and deliver practical data solutions.
• Analyse data and create dashboards and visualisations using tools such as Power BI, Looker and QlikView.
• Turn analysis into clear insights and compelling data stories that help clients make informed decisions.
• Help improve system performance, reliability and cost efficiency through troubleshooting and automation, and keep documentation clear.

Skills

Python

SQL

Spark/Databricks exposure

Delta Lake awareness

Cloud platforms (AWS/Azure/GCP)

Relational databases basics

Airflow/NiFi/Talend familiarity

Power BI/Looker/QlikView dashboards

Git & CI/CD basics

Agile/Scrum awareness

Team/client-facing experience

Tools

Databricks

Power BI

Looker

QlikView

Airflow

Apache NiFi

Talend

Git

CI/CD tooling

Job description

Optima Partners in Edinburgh is seeking a Data Engineer to help design, build and maintain reliable data pipelines for clients across diverse sectors. This early-career role suits someone with up to two years commercial experience or equivalent project work who wants to develop technical and consulting skills.

You'll work alongside data engineers, data scientists and product specialists, with mentoring and hands-on training to grow in modern data platforms, dashboards and data storytelling,