Data Engineer - AI-Driven Big Data Pipelines

Kainos; Belfast City District; On-site; GBP 28,000 - 51,000; Full time

Job summary

Kainos in the United Kingdom is seeking a software engineer to develop data processing software for Big Data deployments, across the full software lifecycle from design to deployment, collaborating with Architects to meet non-functional requirements.

You will mentor teammates, advise on estimates, and work with operations to ensure readiness. Familiarity with AWS, Azure, GCP, Databricks, Kafka, and CDC is valued, and a keen interest in AI technologies is encouraged.

Qualifications

• Strong software development experience in one of Java, Scala, or Python.
• Software development experience with data-processing platforms from vendors such as AWS, Azure, GCP, Databricks.
• Experience of developing substantial components for large-scale data processing solutions and deploying into a production environment.
• Proficient in SQL and SQL extensions for analytical queries.
• Solid understanding of ETL/ELT data processing pipelines and design patterns.
• Aware of key features and pitfalls of distributed data processing frameworks, data stores and data serialisation formats.
• Able to write quality, testable code and has experience of automated testing.
• Experience with Continuous Integration and Continuous Deployment techniques.
• A keen interest in AI Technologies.
• (Desirable) Experience of performance tuning.
• (Desirable) Experience of data visualisation and complex data transformations.
• (Desirable) Experience with streaming and event-processing architectures including technologies such as Kafka and change-data-capture (CDC) products.
• (Desirable) Expertise in continuous improvement and sharing input on data best practice.
• (Desirable) Practical experience with AI technologies, tools, processes and delivery.

Responsibilities

• Working to develop data processing software primarily for deployment in Big Data technologies, encompassing the full software lifecycle including design, code, test, and defect resolution.
• Collaborating with Architects and Lead Engineers to ensure the software supports non-functional needs.
• Working with colleagues to resolve implementation challenges and ensuring code quality and maintainability remains high, leading by example in code quality.
• Collaborating with operations teams to ensure operational readiness.
• Advising customers and managers on the estimated effort and technical implications of user stories and user journeys.
• Coaching and mentoring team members.

Skills

Java

Scala

Python

Tools

AWS

Azure

GCP

Databricks

SQL

Kafka

Job description

Kainos in the United Kingdom is seeking a software engineer to develop data processing software for Big Data deployments, across the full software lifecycle from design to deployment, collaborating with Architects to meet non-functional requirements.

You will mentor teammates, advise on estimates, and work with operations to ensure readiness. Familiarity with AWS, Azure, GCP, Databricks, Kafka, and CDC is valued, and a keen interest in AI technologies is encouraged.