Core Data Engineer

Optima Partners; City of Edinburgh; On-site; GBP 60,000 - 85,000; Full time

Job summary

Optima Partners is seeking a Core Data Engineer to design, build, and optimise data pipelines, architectures, and workflows for a diverse client base in Edinburgh, UK. You will work in cross-functional teams and contribute to scalable data solutions while adopting best practices and emerging trends in data engineering.

You will collaborate with data scientists, analysts, and stakeholders, delivering robust ETL/ELT processes, data integration, and documentation.

Qualifications

• 2+ years of data engineering experience.
• Proven ability to deliver data solutions in consulting or client-facing environments is a plus.
• Experience with Agile or Scrum methodologies is beneficial.

Responsibilities

• Data Pipeline Development: Design and implement robust, scalable data pipelines to collect, transform, and integrate data from various sources.

Skills

Python

Java

Scala

Databricks

Apache Spark

Hadoop

AWS

Azure

GCP

PostgreSQL

SQL Server

Airflow

NiFi

Talend

Snowflake

Redshift

BigQuery

Git

Docker

CI/CD

2+ years data engineering

Consulting experience

Problem-solving

Communication

Teamwork

Data pipelines

Tools

Databricks

Job description

The Role

As a Core Data Engineer, you will play a pivotal role in designing, building, and optimising data pipelines, architectures, and workflows for our diverse client base. You’ll work within dynamic, multi-disciplinary project teams to implement scalable data solutions while adhering to best practices and emerging trends in data engineering.

This is an exciting and valuable opportunity for an enthusiastic, earlier-stage data engineer to advance their career across a variety of commercial sectors. At Optima, you’ll work within cross-functional teams, collaborating with experts in different fields and gaining exposure to varied business challenges.

Optima provides an inclusive space for professional development, with a focus on continuous learning and knowledge sharing. Engineering at Optima is a fast-growing team with strong partnerships in the wider data science community.

With the chance to work alongside seasoned engineers, scientists, data architects, and product specialists, you’ll be contributing to our collective effort to stay at the forefront of a fast-paced, data-driven world. Moreover, there are chances to learn and grow as an engineer in different disciplines, as well as a clear path to career development and promotion within the role.

Key Responsibilities

• Data Pipeline Development: Design and implement robust, scalable, and efficient data pipelines to collect, transform, and integrate data from various sources.

• Data Architecture: Develop and optimise data architectures, including data warehouses, data lakes, and other storage solutions.

• ETL/ELT Processes: Create and maintain reliable ETL/ELT workflows that ensure data quality and accessibility.

• Collaboration: Work closely with data scientists, analysts, and client stakeholders to understand requirements and deliver impactful solutions.

• Performance Optimisation: Monitor and enhance the performance of data systems to ensure minimal downtime and fast query responses.

• Automation and Tools: Identify opportunities for automation and recommend tools to improve data engineering workflows.

• Documentation: Maintain detailed technical documentation for all solutions and processes.

Technical Skills

• Programming: Proficiency in Python, Java, Scala, or similar languages.

• Big Data Technologies: Hands-on experience with big data tools e.g. (Databricks, Apache Spark, Hadoop).

• Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks.

• Expertise in relational databases (e.g. postgres, sql server)

• Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend.

• Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design.

• Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment.

Experience

• 2+ years of experience in data engineering or a similar role.

• Proven track record of delivering data solutions in a consulting or client-facing environment is a plus.

• Experience with Agile or Scrum methodologies is beneficial.

Personal Qualities

• Strong problem-solving and analytical thinking abilities.

• Excellent communication skills to explain complex technical concepts to non-technical stakeholders.

• A team-oriented mindset with a focus on collaboration and adaptability

The Company

Optima Partners is an advanced data and business consultancy headquartered in Edinburgh, UK. We are a practitioner-led organisation that collaborates with top consumer brands to drive transformation and foster customer-centricity through our expertise in customer strategy, innovative design, and advanced data science and engineering.

We help our clients get closer to their customers, to truly understand them, and deliver exactly the right products, engagement, and experiences across all channels and interactions. We specialise in unlocking latent value within organizations through a three-pronged approach that focuses on identifying and enhancing value for customers, within the customer base, and within the business itself. In doing so, we foster sustainable value, paving the way for consistent business growth.

We work with leading consumer brands to tackle and overcome complex business and customer problems to drive transformation and champion customer-centric agendas. We are proud to include some of the leading UK and global brands among our current clients such as Lloyds Banking Group, NatWest Group, Bank of Ireland, Nationwide, Aviva, Biogen, Eon Next, OVO, Virgin Media O2, BT, HMD Global, Centrica, and GSK. We are obsessive about delivering value for our clients and work in a collaborative, engaged, and creative way with our colleagues and clients. We strive to support the transition of knowledge and capability into strategic teams.