Senior Solution Architect - Big Data & Streaming Platforms

Softest Consulting Services Limited; Dublin; On-site; EUR 120,000 - 180,000; Full time

Job summary

Softest Consulting Services Limited is seeking an experienced Solution Architect to own end-to-end architecture for large-scale data, streaming, and integration platforms in a Dublin-based environment.

The role requires senior leadership across multiple workstreams, strong governance experience, and hands-on guidance on Cloudera CDP, Kafka, and cloud-native data services on AWS. Collaboration with governance boards is essential.

Qualifications

• 10+ years’ overall IT experience with data platform, integration or distributed systems delivery.

Responsibilities

• Own the end-to-end solution architecture for large-scale distributed data, streaming and data integration solutions.
• Act as the lead architect for Data Solutions distributed data, integration and streaming platforms across initiatives.
• Accountable for solution architecture across the Enterprise Data Lake, Operational Datastore and CDP environment on AWS.
• Design scalable distributed data and streaming solutions across Cloudera, Kafka, API and cloud platforms.
• Provide architectural oversight across solution delivery ensuring consistency and alignment with designs.
• Work with engineering teams to ensure solutions are deliverable, secure, and scalable.
• Review technical designs to ensure alignment with approved architectures.
• Identify risks, dependencies, non-functional requirements and trade-offs.
• Support delivery planning by breaking solutions into iterative increments.
• Represent designs at architecture review boards and governance forums.
• Develop architectural patterns and engineering standards across the data platform.

Skills

Distributed data platforms

Data lake architecture

Kafka

Spark/Flink

API and REST

NoSQL (HBase/Cassandra)

AWS Cloud

DevOps/CI–CD

Data governance

Architecture leadership

Tools

Cloudera CDH/CDP

Java

Spring Boot

REST services

HBase

Cassandra

Hive/Impala

AWS

Spark

Flink

Job description

• 10+ years’ overall IT experience, with significant experience in data platform, integration, or distributed systems delivery.
• 5+ years’ experience in solution architecture, technical architecture, or lead design roles.
• Proven experience architecting streaming or real-time data applications using Kafka or equivalent event-streaming technologies.
• Experience with Kafka consumer architecture, including consumer groups, offsets, retries, dead-letter handling, ordering, partitioning, and scalability.
• Proven experience designing solutions on Cloudera CDH/CDP, or substantial experience with equivalent enterprise-scale distributed data platforms.
• Experience designing and delivering large-scale data ingestion, transformation, storage, and consumption patterns.
• Strong understanding of batch and real-time data processing architectures.
• Experience working in complex enterprise environments with multiple systems, vendors, platforms, and governance processes.
• Strong understanding of non-functional requirements including performance, scalability, resilience, security, observability, and operational support.
• Experience providing architectural leadership across multiple workstreams, programmes or delivery teams.
• Experience mentoring architects, senior engineers or technical leads and influencing technical decision-making.

Required Technical Skills

• Distributed data platforms and data lake architecture on Cloudera CDH/CDP or equivalent platforms.
• Event streaming and real-time integration technologies, including Kafka and related messaging patterns.
• Large-scale data processing and transformation frameworks such as Spark, Flink, Hive, Impala or equivalent technologies.
• API and service-based integration architectures using technologies such as Java, Spring Boot, and REST services.
• NoSQL and distributed data storage technologies including HBase, Cassandra, or equivalent platforms.
• Cloud-hosted data platforms and services, preferably on AWS.
• Modern engineering practices including DevOps, CI/CD, observability, resilience, and operational support.
• Financial services or banking experience.
• Experience with regulated, secure, enterprise-scale data platforms.
• Experience presenting to architecture governance boards.
• Experience leading architecture communities, defining standards, or contributing to capability development within an architecture practice.
• Exposure to enterprise data governance, lineage, metadata, and data quality tooling.
• Experience with cloud-native data services on AWS.

What you will do:

• Own the end-to-end solution architecture for large-scale distributed data, streaming, and data integration solutions.
• Act as the lead architect for Data Solutions distributed data, integration and streaming platforms, providing architectural direction, governance, design assurance and technical leadership across initiatives, while influencing the evolution of the target architecture and ensuring alignment with enterprise technology strategy, engineering standards and long-term platform objectives.
• Accountable for solution architecture across the Enterprise Data Lake, Operational Datastore and associated data platform services within the Cloudera CDP environment on AWS.
• Design and document scalable distributed data and streaming solutions across Cloudera, Kafka, API, and cloud platforms, including ingestion, processing, storage, and consumption patterns.
• Provide architectural oversight across solution delivery, ensuring consistency, quality and alignment with approved designs.
• Work with engineering teams to ensure solutions are deliverable, scalable, secure, supportable, and aligned to enterprise architecture principles.
• Review technical designs produced by engineering teams and ensure alignment with the approved solution architecture.
• Identify technical risks, dependencies, assumptions, non-functional requirements, and design trade-offs.
• Support delivery planning by helping break solutions into iterative delivery increments and sprint-level outcomes.
• Represent solution designs at architecture review boards and governance forums, engaging effectively with business stakeholders, delivery teams, and enterprise architects.
• Develop and promote architectural patterns, engineering standards and best practices across the data platform ecosystem.

You are good to apply if you are:

• Irish or EU or British Citizen, or holding a Stamp 4 or Stamp 1G permit.
• Residing in the specified job location currently