Architect (Unity Catalog & Hive Metastore)

MacroHire; Singapore; Remote; SGD 90,000 - 120,000; Full time

Job summary

A leading technology recruitment firm is seeking an experienced Architect specializing in Unity Catalog and Hive Metastore to lead data platform migration projects. The ideal candidate will have over 9 years of hands-on experience with tools like Databricks and cloud platforms, particularly Azure. This role is remote and offers significant responsibility in technical project management and collaboration across teams.

Qualifications

• 9-10 years of hands-on experience in data platform engineering or migration projects.
• Proficiency in Databricks, Unity Catalog, and Hive Metastore.
• Expertise in Azure cloud platform, IAM, and networking setup.

Responsibilities

• Evaluate Hive Metastore state and prepare a migration plan.
• Configure cloud resources like storage accounts and IAM roles.
• Migrate tables and validate data integrity post-migration.

Skills

Data platform engineering

Migration projects

Databricks

Unity Catalog

Hive Metastore

Cloud platforms (Azure)

Documentation

Cross-functional collaboration

Tools

UCX assessment tools

BI/reporting tools

Job description

Architect (Unity Catalog & Hive Metastore)

You have a job opportunity as an Architect-1/ Data Engineer/ Migration Specialist for a contract duration of 6-7 months (ext), working remotely in UK Shift. Let's dive into the qualifications, skills, responsibilities, and role and responsibilities for this position:Qualifications & Skills:- You should possess 9-10 years of hands-on experience in data platform engineering or migration projects.- Proficiency in tools such as Databricks, Unity Catalog, Hive Metastore, and UCX assessment tools is essential.- Cloud platform expertise, specifically in Azure, including IAM and networking setup, is required.- Familiarity with BI/reporting tools integration and security policy governance is a plus.- Strong documentation and cross-functional collaboration skills are necessary.Good to have:- Experience with cluster policy design and UC-enabled environments.- Knowledge of ACL mapping strategies and data governance best practices.- Ability to create migration dashboards or reports for leadership visibility.Role & Responsibilities:HMS Assessment & Migration Planning:- Evaluate Hive Metastore (HMS) state, migration complexity, and readiness using UCX Assessment Tool.- Identify external and managed tables, BI dependencies, and environmental separation needs.- Prepare a detailed migration plan and finalize a migration runbook.Platform Setup:- Configure cloud resources like storage accounts, IAM roles, and networking.- Set up Unity Catalog Metastore(s) and assign workspace mappings.- Establish and validate admin roles including Account, Metastore, and Workspace Admins.Unity Catalog Design & Cluster Enablement:- Inventory and categorize data assets including catalogs, schemas, tables, views, and files.- Design catalog/schema hierarchy aligned with business units or domains.- Develop ACL/permission migration strategy and validate cluster policies for UC enablement.Migration Execution:- Use UCX tools to migrate tables and views from HMS to Unity Catalog.- Migrate and validate ACLs and permissions using audit scripts.- Update BI tool configurations and modify application connection strings as required.- Test job cluster policies for post-migration functionality.Post-Migration Validation & Handover:- Verify data integrity, job executions, BI report accuracy, and access control mechanisms.- Document rollback procedures and highlight unresolved findings.- Conduct handover workshops and knowledge transfer sessions with operational stakeholders.This role requires a blend of technical expertise, project management skills, attention to detail, and effective communication for successful execution of data platform migration projects.,

Sign-in & see how your skills match this job

• Architect (Unity Catalog & Hive Metastore)

Find Your perfect Job

Sign-in & Get noticed by top recruiters and get hired fast