Cloud Data Administrator

Affinity; Vancouver; On-site; CAD 100,000 - 160,000; Full time

Benefits offered by this job

Affinity Earn

Job summary

Affinity is seeking a skilled Cloud Data Administrator to join our Vancouver Data Operations team on-site. You will manage an Azure-based data platform with Databricks, Unity Catalog, and ADF, ensuring governance, security, and scalable data workflows.

You will design pipelines, optimize Delta Lake, and collaborate with data scientists and BI analysts while advancing CI/CD for data workloads.

Qualifications

• Bachelor's degree in computer science, information systems, or a related field.
• 4+ years of experience in cloud data engineering, data platform, or analytics engineering.
• Familiarity with data governance, security principles, and data quality best practices.
• Azure Data Engineer certifications are desired.
• Experience with Azure APIM is nice to have.

Responsibilities

• Set-up and monitor job pipelines and workflows to maximize uptime.
• Administer Databricks environments including Unity Catalog for governance and security.
• Support deployment, scheduling, and monitoring of data workflows in Databricks and ADF.
• Implement CI/CD and operational monitoring for pipeline deployments.
• Implement and manage Delta Lake for reliable data operations.
• Design, develop, and optimize scalable data pipelines.
• Collaborate with business and data teams to model data for analytics.
• Translate technical solutions and document for stakeholders.

Skills

PySpark

Spark SQL

Delta Lake

Databricks Workflows

REST API Ingestion

JSON Handling

API Integration

Databricks on Azure

Cluster Management

ADF Pipelines

Unity Catalog

Data Modeling

Data Validation

CI/CD

Git/GitHub

Terraform

Azure APIM

Azure Databricks

Education

Bachelor's degree in CS / IT / related field

Tools

Azure Data Factory

Unity Catalog

Delta Lake

Azure DevOps

GitHub Actions

Terraform

Azure APIM

Azure Databricks

Job description

We are seeking a highly skilled and versatile Cloud Data Administrator to join our Data Operations team. Reporting to the Manager, Data Operations, the Cloud Data Administrator plays a key role in the administration and support of our Azure-based data platform, with a particular focus on Databricks, data pipeline orchestration using tools like Azure Data Factory (ADF), and environment management using Unity Catalog. A strong foundation in data engineering, cloud data administration, and data governance is essential.

This is a full-time role located on-site (no remote) in Vancouver, BC.

Responsibilities:

Platform Administration and Monitoring:

• Set-up and monitor job pipelines and workflows, troubleshooting failures as necessary to ensure maximum uptime.
• Administer Databricks environments, including user access, clusters, and Unity Catalog for data lineage, governance, and security.
• Support the deployment, scheduling, and monitoring of data workflows and jobs in Databricks and ADF.
• Implement best practices for CI/CD, version control, and operational monitoring for pipeline deployments.
• Implement and manage Delta Lake to ensure reliable, performant, and ACID-compliant data operations.

Security and access management:

• Experience administering secure access to Azure Databricks and related data services using RBAC, managed identities, service principals, and Azure Key Vault
• Understanding of data security, credential management, encryption, secret handling, and least-privilege access principles
• Familiarity with Unity Catalog access controls, data governance, auditing, and compliance best practices

Data Modeling and Integration:

• Design, develop, and optimize scalable data pipelines using Azure Databricks and ADF.
• Collaborate with business and data engineering teams to design data models that support analytics and reporting use cases.
• Support integration of data from multiple sources into the enterprise data lake and data warehouse.
• Configure API calls to utilize an Azure APIM platform.
• Maintain and enhance data quality, structure, and performance within the Lakehouse and warehouse architecture.

Collaboration and Stakeholder Engagement:

• Work cross-functionally with business units, data scientists, BI analysts, and other stakeholders to understand data requirements.
• Translate technical solutions into business-friendly language and deliver clear documentation and training when required.

Required Qualifications:

Apache Spark (on Databricks)

• Proficient in PySpark and spark SQL
• Spark optimization techniques (caching, partitioning, broadcast joins)
• Writing and scheduling notebooks/jobs in Databricks
• Understanding of Delta Lake architecture and features
• Working with Databricks Workflows (pipelines and job orchestration)

SQL/Python Programming

• Handling JSON, XML, and other semi-structured formats
• Experience with API integration using requests, http, etc.
• Error handling and logging API Ingestion
• Designing and implementing ingestion pipelines for RESTful API
• Transforming and loading JSON responses to Spark tables
• Databricks on Azure
• Cluster configuration and management
• Unity Catalog features (optional but good to have)
• Creating and managing pipelines for orchestration
• Linked services and datasets for ADLS, Databricks, SQL Server
• Parameterized and dynamic ADF pipelines
• Triggering Databricks notebooks from ADF

Data Engineering Foundations

• Data modeling and warehousing concepts
• Data validation and quality checks
• Working with structured and semi-structured data (JSON, Parquet, Avro)

DevOps & CI/CD

• Git/GitHub for version control
• CI/CD using Azure DevOps or GitHub Actions for Databricks jobs Infrastructure-as-code (Terraform for Databricks or ADF)

Additional Requirements:

• Bachelor's degree in computer science, information systems, or a related field.
• 4+ years of experience in cloud data engineering, data platform, or analytics engineering role.
• Familiarity with data governance, security principles, and data quality best practices.
• Excellent analytical thinking and problem-solving skills.
• Strong communication skills and ability to work collaboratively with technical and non-technical stakeholders.
• Microsoft certifications in Azure Data Engineer, Power Platform, or related field is desired
• Experience with Azure APIM is nice to have
• Knowledge of enterprise data architecture and data warehouse principles (e.g., dimensional modeling) an asset

Affinity Earn:

Know someone who's great for this, or any of our open roles? Earn up to $4,000/year for each successful referral through Affinity Earn. You can also earn up to $50,000 for helping us find new clients. Learn about our referral program at https://affinity-group.ca/earn/ or browse our jobs & follow us at https://www.linkedin.com/company/affinity-staffing/jobs/

About Affinity:

Affinity Group is a technology and business consulting and services company. We believe in creating long term relationships between clients and consultants that foster a mutually beneficial partnership. Affinity is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment is decided on the basis of qualifications, merit and business need.

For more information on Affinity, please visit www.affinity-group.ca