Power Platform Architect

Themesoft Inc; États-Unis; À distance; USD 152 000 - 248 000; Plein temps

Résumé du poste

The Platform Architect role at Themesoft Inc seeks an expert to design and oversee Azure Databricks platforms for a US-based remote contract. You will implement Unity Catalog, secure identity integration, and scalable data workflows to empower data teams and analytics initiatives.

You will lead cross-functional efforts with data engineers, security, and operations, ensuring robust governance, performance tuning, and cost optimization across Databricks workloads.

Qualifications

• 10+ years of platform architecture experience.
• Expertise in Azure Databricks, Unity Catalog, Delta Lake.
• Strong security, IAM, and SCIM integration knowledge.
• Proven ability to design secure, scalable data platforms.
• Excellent collaboration with data engineers, security and operations teams.

Responsabilités

• Architect and oversee Azure Databricks workspace deployments with private endpoints.
• Define cluster policies, DBR versions, auto-scaling and cost controls.
• Design Unity Catalog schemas, FGAC and data lineage.
• Implement Entra ID integration, SCIM provisioning, and secret management.
• Secure networking: VNet, private endpoints, NSGs, and DNS routing.
• Collaborate with stakeholders and mentor teams on best practices.
• Establish monitoring, DR, and CI/CD pipelines for Databricks assets.

Connaissances

Databricks Expertise

Security & IAM

Networking

Scripting/Automation

CI/CD

Troubleshooting

Communication

Leadership

Outils

Azure Databricks

Azure DevOps

GitHub Actions

PowerShell

Azure CLI

Description du poste

Role: Platform Architect with Databricks

Location: United States – Remote

Mode: Contract

Experience: 10+yrs

Responsibilities:

• Design & Deployment: Architect and oversee the deployment of Azure Databricks workspaces, including VNet Injection, secure cluster connectivity (no public IP), and private endpoints.
• Configuration Management: Define and manage Databricks cluster policies, instance types, Databricks Runtime (DBR) versions, and auto-scaling configurations.
• Performance Tuning: Monitor and optimize Databricks workspace and job performance, advising on cluster sizing, Spark configurations, and Delta Lake optimizations.
• Capacity Planning: Forecast and plan for future Databricks resource needs based on platform usage trends.
• Cost Optimization: Identify and implement strategies to optimize Azure Databricks costs (e.g., proper cluster sizing, auto-termination, spot instances, reserved instances).
• Troubleshooting: Provide expert-level troubleshooting for complex Databricks issues, spanning cluster failures, job errors, performance bottlenecks, and connectivity problems.

2. Data Governance & Unity Catalog:

• Unity Catalog Design: Architect and implement the Unity Catalog metastore, including the setup of catalogs, schemas, volumes, and external locations.
• Fine-Grained Access Control (FGAC): Define and enforce data access policies within Unity Catalog, securing tables, views, and managed objects at row and column levels where required.
• Data Lineage & Audit: Leverage Unity Catalog capabilities for data lineage tracking and support auditing requirements for data access.
• Managed Tables & Volumes: Advise and govern the usage of managed and external tables/volumes within Unity Catalog.

3. Security & Access Management (Entra ID, SCIM, Key Vault, ACLs):

• *Azure Entra ID Integration: Design and implement seamless integration of Azure Databricks with Azure Entra ID (formerly Azure Active Directory) for Single Sign-On (SSO) and user/group synchronization.
• SCIM (System for Cross-domain Identity Management): Configure and manage SCIM provisioning for automated user and group lifecycle management between Entra ID and Databricks.
• Azure Key Vault Integration: Architect and implement secure secret management for Databricks using Azure Key Vault-backed secret scopes for notebooks, jobs, and applications. Advise on credential passthrough configurations.
• Access Control Lists (ACLs): Define and enforce ACLs on Azure Data Lake Storage Gen2 (ADLS Gen2) for Databricks service principals and user access. Implement table-level, view-level, and object-level ACLs within Unity Catalog and non-Unity Catalog environments.
• Compliance & Auditing: Ensure Databricks configurations meet organizational security and compliance standards. Assist in auditing access logs and activity.

4. Networking & Connectivity:

• Secure Network Design: Architect and implement secure network connectivity for Azure Databricks, including VNet peering, private endpoints, and network security groups (NSGs).
• Firewall Rules: Define and manage firewall rules to control inbound and outbound traffic for Databricks workspaces, ensuring secure communication with other Azure services and on-premises resources.
• DNS & Routing: Ensure proper DNS resolution and routing for Databricks components and integrated services.
• ADF Integration Patterns: Design and implement robust integration patterns between Azure Data Factory (ADF) and Azure Databricks, including notebook activity execution, data movement, and orchestrating complex data pipelines.
• Automated Deployments: Work with DevOps teams to define and implement CI/CD pipelines for Databricks assets, notebooks, jobs, and configurations using ADF or Azure DevOps.
• API Management: Leverage Databricks REST APIs for automation and integration with other enterprise systems as needed.
• Storage Account Design: Advise on the design and optimal configuration of Azure Data Lake Storage Gen2 accounts for Databricks workloads, including hierarchical namespaces, access tiers, and lifecycle management policies.
• Mount Points & Credential Passthrough: Guide and troubleshoot secure access patterns to ADLS Gen2 from Databricks, including mount points (legacy) and direct access via Entra ID credential passthrough or service principals.

7. Operations, Monitoring & Reliability:

• Monitoring & Alerting: Implement comprehensive monitoring for Databricks workspaces using Azure Monitor, Log Analytics, and Databricks native logging, setting up alerts for critical events (e.g., cluster failures, job errors, resource utilization).
• Disaster Recovery & Business Continuity: Contribute to and validate disaster recovery and business continuity plans for the Databricks platform.
• Patching & Upgrades: Plan and oversee Databricks Runtime (DBR) upgrades and other platform-level maintenance activities.
• Service Health: Monitor Azure service health and ensure proactive communication of any known issues impacting Databricks.

8. Documentation & Best Practices:

• Architectural Documentation: Create and maintain comprehensive architectural diagrams, design documents, and operational runbooks for the Databricks platform.
• Best Practices: Define and evangelize best practices for Databricks usage, security, cost optimization, and performance to data engineering and data science teams.
• Knowledge Transfer: Provide training and guidance to internal teams on Databricks capabilities, new features, and platform usage.

9. Collaboration & Stakeholder Management:

• Collaborate closely with data engineers, data scientists, security teams, network teams, and operations teams to align on architectural decisions and resolve issues.
• Act as a subject matter expert for all Databricks-related queries from internal teams.
• Participate in architectural review boards and provide technical leadership.

Required Skills:

• Databricks Specifics: Unity Catalog, Delta Lake, Spark architecture, Databricks Runtime, Workspaces, Clusters, Jobs, Notebooks, Repos, Databricks APIs.
• Security: Strong understanding of cloud security principles, identity and access management (IAM), role-based access control (RBAC), data encryption, network security. Experience with SCIM.
• Networking: Solid understanding of TCP/IP, VPNs, routing, firewalls, and secure network design in Azure.
• Scripting/Automation: Proficiency in Python, Scala, SQL, and PowerShell/Azure CLI for automation and management tasks.
• CI/CD: Experience with CI/CD pipelines for cloud resources and Databricks artifacts (e.g., Azure DevOps, GitHub Actions).
• Problem-Solving: Excellent analytical and problem-solving skills, with the ability to troubleshoot complex, multi-service issues.
• Communication: Strong verbal and written communication skills to articulate complex technical concepts to both technical and non-technical audiences.
• Certifications (Preferred): Microsoft Certified: Azure Solutions Architect Expert, Databricks Certified Data Engineer Professional/Associate.

Work Authorization

Applicants must be legally authorized to work in the United States.

[Themesoft Inc] is an Equal Employment Opportunity employer. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other status protected by applicable federal, state, or local law.