Senior Data Streaming & Infra Engineer – Hybrid Dublin

emagine Group; Dublin; Hybrid; EUR 90,000 - 120,000; Full time

Benefits offered by this job

Hybrid work

Job summary

emagine Group in Dublin is seeking a Senior Data Infrastructure & Streaming Engineer to design, build and support highly available Kafka and Spark platforms across multi-data centre environments. You will automate infrastructure with Terraform, Ansible and Kubernetes, integrate data streams with enterprise systems and ensure security, observability and reliability.

This 12-month contract offers hybrid working arrangements and a chance to work with a global consulting and technology team on

Qualifications

• Strong hands-on experience administering and supporting Apache Kafka in enterprise environments.
• Experience designing, deploying and optimising Apache Spark clusters.
• Experience with Kubernetes, Docker and containerised platforms.
• Strong experience with Infrastructure as Code and automation tools including Terraform and Ansible.
• Strong Linux administration experience (RHEL and/or Ubuntu).
• Experience integrating data platforms with enterprise applications, databases and data lakes.
• Knowledge of security, authentication and access management technologies including LDAP, Kerberos, OAuth2 and TLS.
• Experience implementing monitoring and observability solutions using tools such as Prometheus, Grafana and ELK.
• Strong understanding of high-availability, disaster recovery and performance tuning principles.

Responsibilities

• Design, build and support highly available Apache Kafka and Apache Spark platforms across multi-data centre environments.
• Develop and maintain real-time data streaming and large-scale data processing solutions.
• Automate infrastructure deployment and configuration using Terraform, Ansible and Kubernetes.
• Integrate streaming platforms with enterprise systems, databases, data lakes and microservices.
• Implement security controls including authentication, authorisation and encryption.
• Monitor, maintain and optimise Kafka and Spark clusters to ensure performance, scalability and reliability.
• Implement observability solutions using monitoring, logging and alerting tools.
• Troubleshoot platform issues and support incident resolution and root cause analysis.
• Collaborate with infrastructure, security and engineering teams to deliver enterprise-grade data platforms.

Skills

Apache Kafka

Apache Spark

Kubernetes

Terraform

Ansible

Linux administration

Security best practices

Observability tools

Data integration

Tools

Docker

Proxmox

Job description

emagine Group in Dublin is seeking a Senior Data Infrastructure & Streaming Engineer to design, build and support highly available Kafka and Spark platforms across multi-data centre environments. You will automate infrastructure with Terraform, Ansible and Kubernetes, integrate data streams with enterprise systems and ensure security, observability and reliability.

This 12-month contract offers hybrid working arrangements and a chance to work with a global consulting and technology team on