Big Data Architect

MBR Partners; London; On-site; GBP 60,000 - 100,000; Full time

Job summary

An innovative firm is seeking an AiOps Architect to design and support a Cloud Native Big Data Analytics infrastructure. This role is crucial for enabling AI/ML workflows and involves working with cutting-edge technologies to handle massive data throughput. The ideal candidate will have extensive experience in software development, particularly in back-end data processing, and will leverage their knowledge of Big Data open-source technologies. This position offers the opportunity to shape the future of data analytics in a dynamic environment, making a significant impact on the company's success.

Qualifications

• Experience in back-end data processing and data lakehouse architectures.
• Hands-on with Big Data technologies and data governance tools.

Responsibilities

• Lead the vision for a coherent Big Data Analytics framework for AI/ML.
• Design and validate architecture components supporting the product roadmap.

Skills

Software Development

Back-end Data Processing

Data Lakehouse Architectures

Big Data Technologies

Data Governance Tools

Kubernetes

OpenShift

Tools

Apache Airflow

Kafka

Pekko

Spark

Spark Structured Streaming

Delta Lake

AWS Athena

Trino

MongoDB

AWS S3

MinIO S3

Job description

Job Opportunity: AiOps Architect

We are headhunting for an AiOps Architect and would like to see if you have any interest or could recommend someone for this role.

Our client is a mid-sized leader in AiOps Telecoms software products related to network management and service assurance.

The role involves designing and supporting the implementation of the client's Cloud Native Big Data Analytics infrastructure layer. This infrastructure supports AI/ML workflows, enabling real-time and batch processing for training and inference, handling data throughput of terabytes per hour.

Ideal Candidate Background

• Software development experience, especially in back-end data processing and data lakehouse architectures.
• Hands-on experience with Big Data open-source technologies such as Apache Airflow, Kafka, Pekko, Spark & Spark Structured Streaming, Delta Lake, AWS Athena, Trino, MongoDB, AWS S3, MinIO S3.
• Proven success in setting up data governance tools and processes (schema registry, data lineage, access control), data pipelines for model training and inference, and working with Kubernetes or OpenShift in Big Data analytics contexts.

Role Responsibilities

• Lead the vision for a coherent, future-proof Big Data Analytics framework for AI/ML.
• Leverage GenAI technologies to enhance data processing capabilities.
• Design and validate architecture components supporting the product roadmap, including testing, performance, and scalability.
• Develop project designs, communicate architecture to teams, and oversee development reviews.
• Produce technology watch reports and support the selection of third-party components.
• Educate teams on analytics best practices, review designs and code, and support technical queries related to Big Data Analytics.
• Conduct feasibility studies and high-level costings aligned with long-term strategies.

Application Process

You can apply by uploading your resume or completing your profile manually. Supported file formats include doc, docx, txt, rtf, and pdf, smaller than 10MB.

Required information includes your full name, email, and experience details.

By submitting, you agree to data processing in accordance with GDPR policies.