Senior Data Infrastructure & Streaming Engineer

emagine Group; Dublin; Hybrid; EUR 90,000 - 120,000; Full time

Benefits offered by this job

Hybrid work

Job summary

emagine Group in Dublin is seeking a Senior Data Infrastructure & Streaming Engineer to design, build and support highly available Kafka and Spark platforms across multi-data centre environments. You will automate infrastructure with Terraform, Ansible and Kubernetes, integrate data streams with enterprise systems and ensure security, observability and reliability.

This 12-month contract offers hybrid working arrangements and a chance to work with a global consulting and technology team on

Qualifications

• Strong hands-on experience administering and supporting Apache Kafka in enterprise environments.
• Experience designing, deploying and optimising Apache Spark clusters.
• Experience with Kubernetes, Docker and containerised platforms.
• Strong experience with Infrastructure as Code and automation tools including Terraform and Ansible.
• Strong Linux administration experience (RHEL and/or Ubuntu).
• Experience integrating data platforms with enterprise applications, databases and data lakes.
• Knowledge of security, authentication and access management technologies including LDAP, Kerberos, OAuth2 and TLS.
• Experience implementing monitoring and observability solutions using tools such as Prometheus, Grafana and ELK.
• Strong understanding of high-availability, disaster recovery and performance tuning principles.

Responsibilities

• Design, build and support highly available Apache Kafka and Apache Spark platforms across multi-data centre environments.
• Develop and maintain real-time data streaming and large-scale data processing solutions.
• Automate infrastructure deployment and configuration using Terraform, Ansible and Kubernetes.
• Integrate streaming platforms with enterprise systems, databases, data lakes and microservices.
• Implement security controls including authentication, authorisation and encryption.
• Monitor, maintain and optimise Kafka and Spark clusters to ensure performance, scalability and reliability.
• Implement observability solutions using monitoring, logging and alerting tools.
• Troubleshoot platform issues and support incident resolution and root cause analysis.
• Collaborate with infrastructure, security and engineering teams to deliver enterprise-grade data platforms.

Skills

Apache Kafka

Apache Spark

Kubernetes

Terraform

Ansible

Linux administration

Security best practices

Observability tools

Data integration

Tools

Docker

Proxmox

Job description

Want to make an impact? We offer that

Due to continued growth, we are currently hiring a Senior Data Infrastructure & Streaming Engineer to join our team.

emagine is a dynamic, fast-growing international consulting and technology company, founded on real-world experience, focused on custom-fit solutions. We have the proven people and expertise to complete complex projects, while delivering lasting knowledge-transfer benefits to empower our partners for the next challenge.

Are you intrigued? Do you want to learn more?

A snapshot of your key responsibilities as a Senior Data Infrastructure & Streaming Engineer would be:

• Design, build and support highly available Apache Kafka and Apache Spark platforms across multi-data centre environments.
• Develop and maintain real-time data streaming and large-scale data processing solutions.
• Automate infrastructure deployment and configuration using Terraform, Ansible and Kubernetes.
• Integrate streaming platforms with enterprise systems, databases, data lakes and microservices.
• Implement security controls including authentication, authorisation and encryption.
• Monitor, maintain and optimise Kafka and Spark clusters to ensure performance, scalability and reliability.
• Implement observability solutions using monitoring, logging and alerting tools.
• Troubleshoot platform issues and support incident resolution and root cause analysis.
• Collaborate with infrastructure, security and engineering teams to deliver enterprise-grade data platforms.

Requirements

• Strong hands-on experience administering and supporting Apache Kafka in enterprise environments.
• Proven experience designing, deploying and optimising Apache Spark clusters and workloads.
• Experience with Kubernetes, Docker and containerised platforms.
• Strong experience with Infrastructure as Code and automation tools including Terraform and Ansible.
• Strong Linux administration experience (RHEL and/or Ubuntu).
• Experience integrating data platforms with enterprise applications, databases and data lakes.
• Knowledge of security, authentication and access management technologies including LDAP, Kerberos, OAuth2 and TLS.
• Experience implementing monitoring and observability solutions using tools such as Prometheus, Grafana and ELK.
• Strong understanding of high-availability, disaster recovery and performance tuning principles.

Desirable Experience

• Experience supporting multi-data centre or geographically distributed environments.
• Experience with Proxmox, VMware or other virtualised infrastructure platforms.
• Knowledge of enterprise storage technologies including SAN and NVMe storage.
• Experience with Keycloak, FreeIPA or Active Directory integrations.
• Experience working within large-scale data, analytics or regulated environments.

So are you ready to join our team?

It’s important to remember, emagine is an Equal Employment Opportunity Employer; we value bringing together individuals from diverse backgrounds to develop innovative solutions for our customers.

Opportunity type: Contract (12 months initially), Dublin (Hybrid)