Solutions Architect- Databricks

RiDiK (a Subsidiary of CLPS. Nasdaq: CLPS); Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

RiDiK, a subsidiary of CLPS, invites senior data architects to lead end-to-end data architecture projects on the Databricks platform in Singapore. You will design scalable ingestion strategies, implement Delta Lake-based storage, and develop Spark-based processing workflows to support analytics and ML use cases.

You will collaborate with Databricks PS and client teams to ensure governance, security, and compliance while mentoring junior engineers and aligning designs with enterprise standards.

Qualifications

• 8-10 years in data architecture, engineering, or analytics roles.
• 3-5 years hands-on with Databricks platform.
• Databricks Certified Associate Developer for Apache Spark.
• Proven Unity Catalog implementation in enterprise environments.

Responsibilities

• Collaborate with Databricks Professional Services and customers on end-to-end data architecture on the Databricks platform.
• Design and implement data ingestion strategies from diverse source systems.
• Architect scalable storage using Delta Lake and lakehouse architecture.
• Develop data processing frameworks with Apache Spark and Databricks workflows.
• Create data consumption layers for analytics, reporting, and ML use cases.
• Lead technical discussions and provide guidance to development teams.

Skills

Databricks Platform

Unity Catalog

Data Modelling

Delta Lake

Apache Spark

Python/SQL/Scala

Cloud Platforms

Tools

Databricks

Job description

Duration: 1 year contract subject to renewal

End-to-End Data Architecture

• Collaborate with Databricks PS and customer
• on comprehensive end-to-end data architecture on Databricks platform
• Design and implement data ingestion strategies from various source systems
• Architect scalable storage solutions leveraging Delta Lake and lakehouse architecture
• Develop efficient data processing frameworks using Apache Spark and Databricks workflows
• Create data consumption layers for analytics, reporting, and machine learning use cases
• Ensure seamless data flow across the entire data lifecycle

Governance, Security & Compliance

• Collaborate with Databricks PS on implementing robust governance frameworks
• Design and enforce security policies and access controls within Databricks environments
• Ensure compliance with regulatory requirements and industry standards
• Implement data lineage, audit trails, and monitoring capabilities
• Optimize performance through best practices in cluster configuration and query optimization
• Establish data quality standards and validation processes
• Define business-aligned data models that support organizational objectives
• Design Delta table structures optimized for performance and scalability
• Create dimensional and normalized data models based on business requirements
• Implement medallion architecture (Bronze, Silver, Gold layers) for data refinement
• Develop data schemas that support both analytical and operational workloads
• Document data models and maintain data dictionaries

Technical Leadership & Collaboration

• Lead technical discussions with clients and stakeholders to understand business requirements
• Provide technical guidance to development teams on Databricks best practices
• Work closely with Infrastructure (Infra), Applications (Apps), and Cyber teams for integrated delivery
• Mentor junior architects and engineers on Databricks technologies
• Present solution architectures and technical recommendations to client leadership

Solution Implementation

• Oversee implementation of Databricks solutions from design through deployment
• Ensure solutions meet performance, scalability, and reliability requirements
• Support proof-of-concepts and pilot implementations
• Conduct architecture reviews and ensure alignment with enterprise standards

Key Skills

• Databricks Platform: Proven hands-on experience with Databricks platform including workspace administration, cluster management, and workflow orchestration
• Unity Catalog: Demonstrated experience implementing Unity Catalog for unified governance, including metastore configuration, catalog/schema design, and access control policies
• Data Modelling: Strong expertise in designing and implementing data models, including dimensional modeling, data vault, and lakehouse architectures
• Delta Lake: Deep understanding of Delta Lake features including ACID transactions, time travel, and optimization techniques
• Apache Spark: Proficiency in Spark SQL, DataFrames, and performance tuning
• Programming: Strong skills in Python, SQL, and/or Scala for data processing
• Cloud Platforms: Experience with Azure, AWS, or GCP cloud services

Additional Technical Skills

• Data pipeline development and ETL/ELT processes
• Data governance frameworks and metadata management
• Performance optimization and troubleshooting
• CI/CD practices for data platforms
• Data quality and validation frameworks
• Understanding of data security and compliance requirements

Professional Experience

• Minimum 8-10 years in data architecture, engineering, or analytics roles
• At least 3-5 years of hands-on experience with Databricks platform
• Databricks Certified Associate Developer for Apache Spark
• Proven track record implementing Unity Catalog in enterprise environments
• Experience designing and implementing complex data models for large-scale systems
• Background working with Databricks Professional Services or similar partnerships preferred
• Experience across multiple industry sectors (Public Service, Financial Services, Healthcare, etc.) is advantageous

Certifications (Preferred) - mandatory to have at least one certification

• Data Engineer Professional
• Cloud platform certifications (Azure Data Engineer, AWS Data Analytics, or GCP Data Engineer)
• Relevant data management or analytics certifications