Senior Data Engineer

Kpler; Deutschland; Vor Ort; EUR 70.000 - 100.000; Vollzeit

Zusammenfassung

Kpler, a leader in providing insights for commodities and maritime sectors, seeks a Senior Data Engineer to develop data streaming architectures. The role involves crafting robust data processing systems and ensuring performance excellence, demanding high expertise in Scala and Kafka. Join a dynamic team focused on innovation and continuous improvement.

Qualifikationen

• 5-8 years in data engineering with expertise in data streaming.
• Proficiency in Scala and Apache Kafka required.
• Experience in Test-Driven Development (TDD) and designing high-availability systems.

Aufgaben

• Lead design and deployment of data streaming architectures.
• Collaborate with Product for technical requirements.
• Provide technical leadership and uphold engineering standards.

Kenntnisse

Scala

Apache Kafka

Data Engineering

Software Engineering

Test-Driven Development (TDD)

Distributed Systems Design

Agile Development

Tools

Apache Airflow

Apache Spark

Terraform

GitOps

Jobbeschreibung

At Kpler, we are dedicated to helping our clients navigate complex markets with ease. By simplifying global trade information and providing valuable insights, we empower organisations to make informed decisions in commodities, energy, and maritime sectors.

Since our founding in 2014, we have focused on delivering top-tier intelligence through user-friendly platforms.Our team of over 600 experts from 35+ countries works tirelessly to transform intricate data into actionable strategies, ensuring our clients stay ahead in a dynamic market landscape. Join us to leverage cutting-edge innovation for impactful results and experience unparalleled support on your journey to success.

Data Engineers play a crucial role in creating the architecture and systems needed to collect, store, transform, and access data.

Our team is seeking a Senior Data Engineer with in-depth expertise in data streaming technologies and software development. They design, build, and maintain large-scale processing systems, data pipelines, and infrastructure, with a particular focus on robust data streaming architectures.These systems power mission-critical products and contribute to the technical excellence and continuous improvement of our data platforms.

In this role, you’ll be deeply involved in user stories refinement, product development, and the design of high-performance, resilient, and high-availability distributed systems.You will take full ownership of the features you develop, guiding them from conception and build through to deployment and ensuring their effective operation in production.

Responsibilities

• Lead the design, development, and deployment of robust data streaming architectures and high-performance, resilient data processing systems, primarily using Kafka Streams.
• Collaborate with Product to define technical requirements and deliver high-quality, scalable software solutions.
• Take ownership of system performance, designing for high availability, feature maintenance, issue troubleshooting, and ensuring robust operation of data platforms.
• Uphold and enforce the highest engineering standards, championing a culture of software quality, Software Craftsmanship, constructive critique, and continuous improvement.
• Provide technical leadership within the team, helping to elevate engineering capabilities.
• Deliver well-documented, maintainable code following Test-Driven Development (TDD) principles, ensuring comprehensive unit, integration, and end-to-end testing.
• Demonstrate a strong sense of ownership, autonomy, a proactive approach to problem-solving, and the ability to provide innovative technical solutions, challenging existing approaches with well-reasoned arguments.

Skills and Experience

• At least 5 - 8 years of experience in a data engineering or software engineering role with a focus on data-intensive systems.
• Have confirmed expertise in Scala.
• Strong experience with Apache Kafka and specifically Kafka Streams.
• Strong software engineering capabilities and experience in designing high-availability distributed systems.
• Experienced in Test-Driven Development (TDD) and proficient in writing comprehensive unit, integration, and end-to-end tests.
• Strong collaboration and communication skills, with an ability to articulate and defend technical decisions.
• Demonstrate architectural rigor, a systemic vision, and a proactive, problem-solving mindset.
• Embrace a culture of constructive critique, continuous improvement, and a Software Craftsmanship approach.
• Strong ability to work autonomously and provide innovative technical solutions.
• Experience working with Agile Development methodologies.

Nice to have

• Experience with Airflow or Apache Spark
• Experience using Infrastructure as code tools like Terraform
• Experience with GitOps workflows and Continuous Deployment system

We are a dynamic company dedicated to nurturing connections and innovating solutions to tackle market challenges head-on. If you thrive on customer satisfaction and turning ideas into reality, then you’ve found your ideal destination. Are you ready to embark on this exciting journey with us?

We make things happen

We act decisively and with purpose, going the extra mile.

We buildtogether

We foster relationships and develop creative solutions to address market challenges.

We are here to help

We are accessible and supportive to colleagues and clients with a friendly approach.

Our People Pledge

Don’t meet every single requirement? Research shows that women and people of color are less likely than others to apply if they feel like they don’t match 100% of the job requirements. Don’t let the confidence gap stand in your way, we’d love to hear from you! We understand that experience comes in many different forms and are dedicated to adding new perspectives to the team.

Kpler is committed to providing a fair, inclusive and diverse work-environment. We believe that different perspectives lead to better ideas, and better ideas allow us to better understand the needs and interests of our diverse, global community. We welcome people of different backgrounds, experiences, abilities and perspectives and are an equal opportunity employer.

By applying, I confirm that I have read and accept theStaff Privacy Notice