Senior Data Engineer

Expleo; Derby; On-site; GBP 35,000 - 65,000; Full time

Job summary

A leading company in engineering and consulting seeks a skilled Data Engineer to enhance data migration efforts and implement predictive solutions for jet engines. The role demands strong expertise in Python, SQL, and Azure services, ensuring high quality in data management processes in a team-oriented environment.

Benefits

Collaborative working environment

Expleo Academy for skill development

Competitive company benefits

Qualifications

• Strong experience in Databricks and SQL.
• Expertise in PySpark and data migration processes.
• Excellent verbal and written communication skills.

Responsibilities

• Collaborate on data migration requirements and quality issues.
• Document migration processes and provide updates.
• Design and maintain data migration modules.

Skills

Databricks

SQL

Azure Services

PySpark

Data Migration

Communication

Tools

Azure DevOps

COSMOS DB

Python

Job description

Overview

Expleois a trusted partner for end-to-end, integrated engineering, quality services and management consulting for digital transformation. We help businesses harness unrelenting technological change to successfully deliver innovations that will help them gain a competitive advantage and improve the everyday lives of people around the globe.

We are seeking a talented Data Engineer to join our team. As a Data Engineer, you will work with a variety of technologies including Python, Py-Spark, Pandas, SQL and Azure DevOps. You will be responsible for modeling and implementing predictive maintenance solutions for our products, specifically focusing on jet engines.

Responsibilities

• Collaborate with stakeholders to understand data migration requirements

• Work with diverse data formats and source/target databases

• Design, develop, test, and maintain data migration modules

• Analyse and address data quality issues at both the source and destination

• Participate in peer reviews, design discussions, and knowledge sharing

• Prioritize and resolve issues reported in live systems to ensure minimal downtime

• Document migration processes and provide regular progress updates

Essential skills

• Strong experience with Databricks

• Proficient in SQL for data querying and transformation

• Solid understanding of Azure Services including:

  • Azure Data Factory

  • Azure Storage Containers

  • Lakehouse architecture

• Expertise in PySpark for big data processing

• In-depth knowledge of data migration processes, including ETL and data mapping

• Excellent verbal and written communication skills

Experience

• Experience with CI/CD pipelines for deployment automation

• Familiarity with COSMOS DB

• Proficiency in Python beyond PySpark

• Working knowledge of Azure DevOps tools and practices

Benefits

• Collaborative working environment – we stand shoulder to shoulder with our clients and ourpeers through good times and challenges
• We empower all passionate technology loving professionals by allowing them to expand their skills and take part in inspiring projects
• ExpleoAcademy - enables you to acquire and develop the right skills by delivering a suite of accredited training courses
• Competitive company benefits
• Always working as one team, our people are not afraid to think big and challenge the status quo

• As a Disability Confident Committed Employer we have committed to:
• Ensure our recruitment process is inclusive and accessible
• Communicating and promoting vacancies
• Offering an interview to disabled people who meet the minimum criteria for the job
• Anticipating and providing reasonable adjustments as required
• Supporting any existing employee who acquires a disability or long term health condition, enabling them to stay in work at least one activity that will make a difference for disabled people

“We are an equal opportunities employer and welcome applications from all suitably qualified persons regardless of their race, sex, disability, religion/belief, sexual orientation or age”.

We treat everyone fairly and equitably across the organisation, including providing any additional support and adjustments needed for everyone to thrive

#LI-DS1