Data/AI Architect (Databricks)

Blue Pearl HQ; Johannesburg; On-site; ZAR 1,200,000 - 1,800,000; Full time

Job summary

Blue Pearl HQ in Johannesburg is seeking a senior data/AI architect to plan AI solutions aligned with business objectives and to build the architecture capability across the enterprise. You will design AI architectures on Databricks and cloud-native tech, govern models, and drive guardrails, standards and blueprints.

You will partner with the AI Technology Centre of Excellence to evolve the Databricks AI platform, support enterprise AI and generative AI use cases, and lead architecture reviews,

Qualifications

• Minimum 5 years in technical solution design
• Experience with AI, data engineering and governance
• Databricks platform expertise (Delta Lake, Spark, MLflow)

Responsibilities

• Lead AI solution requirements gathering and align with business objectives.
• Define and refine AI architecture runways with key stakeholders.
• Design and implement enterprise-grade AI architectures on Databricks and cloud-native tech.
• Maintain AI solution artifacts in enterprise repositories and governance.
• Manage AI architecture risk, compliance, and security governance.
• Collaborate with delivery teams and AI engineers to achieve architecture objectives.
• Ensure CI/CD readiness for data solutions and AI deployments.

Skills

Databricks

AI architecture

MLOps

Cloud platforms

SQL

Python

Education

Degree in IT/CS/Engineering

Tools

Delta Lake

Spark

MLflow

Unity Catalog

Job description

The key focus for the senior data/AI architect is to perform planning aligned to key AI solutions, build and participate in the architecture capability building, perform AI architecture and design, manage AI architecture risk and compliance, provide design and build governance and support and communicate and share knowledge around the architecture practices, guardrails, blueprints and standards related to the AI solution design.

A key focus of this role is partnering with the AI Technology Centre of Excellence to build out the organisation's Databricks AI platform and support the delivery of enterprise AI and generative AI use cases.

Planning

• Lead AI solution requirements gathering and ensure alignment with business objectives and constraints.
  
• Define and refine AI architecture runways for intentional architecture with the key stakeholders
  
• Provide input into business cases and costing
  
• Participate and provide AI architectural runway requirements into Programme Increment (PI) Planning
  
• Design and implement enterprise-grade AI architectures leveraging Databricks and cloud-native technologies.
  
• Develop and oversee AI architecture views and ensure alignment with enterprise architecture.
  
• Maintain and oversee the AI solution artifacts in the set enterprise repository and knowledge portals aligned to the rest of the architecture
  
• Manage the AI architecture processes based on the requirements for each architype
  
• Manage change impact of the AI architecture with stakeholders
  
• Develop and participate in the build of the AI architecture practice with embedded architects and engineers including the relevant methods, repository and tools
  
• Manage the AI architecture considering the business, application, information/data and technology viewpoints
  
• Establish, enforce and implement AI standards, guardrails, frameworks, and patterns
  
• Partner with the AI Tech COE to define and evolve the Databricks AI platform architecture, ensuring alignment with enterprise data and AI strategy
  
• Design and implement AI/ML architectures on Databricks, including MLOps pipelines, model lifecycle management, Unity AI Gateway and Unity Catalog governance for AI/ML assets
  

Solution Design

• Lead and review logical and detailed AI architecture
  
• Evaluate and approve AI solution options and technology selections
  
• Select appropriate technology, tools and build for the solution
  
• Oversee and maintain the AI solution blueprints
  
• Drive incremental modernisation initiatives in the delivery area
  
• Design and evaluate architectures for AI and generative AI use cases, including RAG pipelines, vector stores, feature stores, and LLM integration patterns
  

Risk, Governance and Compliance

Identify, assess and mitigate risks at a AI solution architecture level

Ensure and enforce compliance with policies, standards, and regulations

Lead AI architecture reviews and integrate with governance functions

Integrate with other governance and compliance functions to ensure continuity in managing the investment and risk for the organisation pertaining to the solution architectures

Establish and provide AI standards, guidance, and tools to delivery teams.

Implementation and Collaboration

Establish and provide AI solution architectures and tools to the delivery and AI engineering teams

Lead and facilitate collaboration with delivery teams to achieve architecture objectives

Manage and resolve deviations and ensure up-to-date AI solution design documentation

Identify opportunities to optimise delivery of solutions

Oversee and conduct post-implementation reviews

Ensure the AI architecture supports CI/CD pipelines to facilitate rapid and reliable deployment of data solutions

Implement automated testing frameworks for AI solutions to ensure quality and reliability throughout the development lifecycle.

Establish performance monitoring and optimisation practices to ensure AI solutions meet performance benchmarks and can scale as needed.

Integrate robust AI security measures, including encryption, access controls, and regular security audits, into the implementation process.

Communication and Knowledge Sharing

Communicate and advocate up-to-date AI solution architecture views

Communicate the relevant AI standards, practices, guardrails and tools to stakeholders relevant to the solution design

Ensure IT teams are well-informed and trained in architecture requirements

Communicate and collaborate with stakeholders' relevant views on planning, technology assessments, risk, compliance, governance and implementation assessments

Foster collaboration between AI architects, AI engineers, and other IT teams through regular cross-functional meetings and agile ceremonies.

Communicate and maintain up-to-date blueprint designs for key data solutions

Ensure effective participation in the agile ceremonies (PI planning, sprint planning, retrospectives, demos)

Implement regular feedback loops with stakeholders and end-users to continuously improve data solutions based on real-world usage and requirements

Create a culture of knowledge sharing by organising regular workshops, training sessions, and documentation updates to keep all team members informed about the latest AI architecture practices and tools

Requirements

MINIMUM QUALIFICATIONS/EXPERIENCE

• Matric
• Degree or diploma in Information Technology, Computer Science, Engineering OR relevant diploma / degree
• Experience: Requires a minimum of 5 years in a technical/solution design role and a minimum of 7 years relevant IT experience
• Data and AI Experience:Required a minimum of 7 years related experience in AI, data engineering, data modeling and design and data management and governance
• Expert-level proficiency in Databricks, including Delta Lake, Spark, and MLflow.

Proven experience architecting and delivering AI/ML solutions on Databricks, including MLOps, model deployment and monitoring, and Unity Catalog governance for AI/ML assets.

Hands-on experience in large-scale data and AI platform implementation (preferably cloud-based).

ADDITIONAL QUALIFICATIONS/EXPERIENCE (PREFERRED, NOT A REQUIREMENT)

• DAMA-DMBOK
• TOGAF
• ArchiMate
• Cloud Certifications (AWS, Azure)
• Financial Industry Experience
• Certifications in Databricks, AWS ML, AWS Data Engineering or similar.
• Experience with generative AI / LLM architectures (e.g. RAG pipelines, vector databases, AI gateways)
• Databricks Certified Machine Learning Professional or equivalent certification

Data Related Experience:

Big Data and Analytics (e.g., Hadoop, Spark)

Data Warehousing

Master Data Management (MDM)

Data Lakes, Lakehouse, and Data Mesh

Metadata Management

ETL/ELT Processes

Data Privacy and Compliance

Cloud Data Services

Experience with AI cloud platforms (Azure, AWS, or GCP) and associated data services.

Proficiency in SQL, Python, and distributed data processing frameworks.

Familiarity with CI/CD for data pipelines and DevOps practices.

Experience with Lakehouse architecture and real-time streaming solutions

Related attributes and competencies related to architecture:

Critical thinking/problem solving

Teamwork/collaboration

Effective Communication Skills

Leadership skills

Knowledge and experience in architecture domains

Knowledge and experience in architecture methods, frameworks and tools

Solution Design Experience

Agile Knowledge and Experience

Cloud Knowledge and Experience

AI related competencies:

AI architecture principles and methodologies

AI integration technologies and tools

AI management and governance

AI/ML architecture, MLOps, and model lifecycle management knowledge and experience