Senior Databricks Data Platform Engineer (PySpark)

Enterprise Solutions Inc.; Montreal; Hybrid; CAD 120,000 - 170,000; Full time

Job summary

Enterprise Solutions Inc. in Montreal (remote) is seeking a Senior Databricks Data Platform Engineer to own the Databricks Lakehouse platform, codify infrastructure, and drive secure, cost-efficient operations across environments.

You will focus on platform as product, automate CI/CD with Terraform, Databricks Asset Bundles, GitHub Actions, and lead guardrails for reliability, performance, and FinOps, while mentoring engineers and collaborating with cross-functional teams.

Qualifications

• Databricks expertise required.
• Experience with cloud Databricks platforms preferred.
• Familiarity with FinOps and DevOps practices is advantageous.

Responsibilities

• Own architecture, configuration and lifecycle of Databricks environments across dev/test/prod.
• Design and maintain CICD pipelines for Databricks code and config.
• Codify infrastructure with Terraform and ensure version-controlled assets.
• Develop agentic AI workflows for platform operations and guardrails.
• Optimize platform costs and governance while maintaining reliability.
• Collaborate with data teams to set platform standards and drive adoption.

Skills

Databricks

Azure Databricks

Job description

Location: Montreal, QC (REMOTE)

Type: Contract

Role Overview

• As a Senior Databricks Data Platform Engineer you will own the engineering automation and day to day health of our enterprise Databricks Lakehouse platform This is a platform-first role rather than building individual business pipelines you will build the foundation that every data engineer analyst and data scientist at Client depends on including workspaces compute Unity Catalog governance deployment pipelines and the guardrails that keep the platform secure reliable and cost-efficient
• You will treat the platform as a product That means codifying infrastructure and configuration delivering changes through fully automated CICD introducing agentic AI workflows that reduce manual operational effort and continuously driving down compute and storage cost while improving performance You will combine deep hands-on Databricks expertise with a FinOps and DevOps mindset and you will measure your success in platform reliability developer productivity and dollars saved
• This is a highly collaborative role You will partner with data engineering data science security cloud infrastructure and finance teams to set platform standards, guide adoption of new Databricks capabilities and mentor engineers across the organization

About the role

• Own the architecture configuration and lifecycle of Client Databricks environments across development test and production, including workspaces, account-level settings, networking and identity integration
• Administer and evolve Unity Catalog including metastores catalogs schemas, external locations storage credentials and fine-grained access controls
• Design and enforce compute strategy across all-purpose clusters job clusters, serverless compute SQL warehouses instance pools and cluster policies
• Manage platform upgrades Databricks Runtime adoption feature previews and deprecations with minimal disruption to consuming teams
• Build reusable platform templates starter projects and self-service capabilities that let engineering teams onboard quickly and consistently

CICD Platform Automation

• Design build and maintain CICD pipelines for Databricks code and configuration using Databricks Asset Bundles Git folders and Azure DevOps or GitHub Actions
• Codify platform infrastructure with Terraform Databricks and cloud providers ensuring every workspace cluster policy, permission and Unity Catalog object is version-controlled and reproducible
• Implement automated testing code quality gates security scanning and promotion workflows from development through production
• Establish branching release and rollback standards for notebooks jobs Delta Live Tables, Lakeflow pipelines and ML assets
• Automate routine administration such as user and group provisioning SCIM secret management token rotation and environment drift detection

Agentic Workflows AI-Enabled Operations

• Design and deploy agentic workflows that automate platform operations such as job failure triage rootcause analysis cost anomaly investigation and remediation recommendations
• Build AI agents and tools on Databricks Mosaic AI Agent Framework Model Serving Vector Search and integrate them with enterprise systems through Model Context Protocol MCP
• Use Copilot and Anthropic models to accelerate infrastructure-as-code pipeline development code review and documentation
• Establish guardrails evaluation, observability and human-in-the-loop controls so agentic automation is safe auditable and trusted
• Partner with data science and engineering teams to provide a governed platform foundation for GenAI RAG and ML workloads

Platform Cost Optimization FinOps

• Own Databricks cost visibility and accountability using system tables billing usage compute and query history tagging standards and chargeback or showback reporting
• Identify and deliver savings through cluster rightsizing, auto-termination, autoscaling tuning spot and reserved capacity strategies Photon and serverless evaluation and workload consolidation
• Enforce cost guardrails through cluster policies budget s and compute governance that prevent waste without slowing teams down
• Build dashboards and automated s that surface cost anomalies idle resources and optimization opportunities to engineering and finance stakeholders
• Partner with finance and procurement on DBU commitments forecasting and capacity planning

Storage Performance Optimization

• Define and enforce Delta Lake storage standards including file sizing liquid clustering Zordering partitioning strategy and deletion vectors
• Automate table maintenance using predictive optimization OPTIMIZE VACUUM and retention policies to control storage growth and improve query

Skills

Mandatory Skills : Databricks

Good to Have Skills : Azure Databricks

We are an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees