Senior Databricks Data Engineer

49 Solutions; Ottawa; Hybrid; CAD 120.000 - 170.000; Vollzeit

Zusammenfassung

49 Solutions is seeking a Senior Databricks Data Engineer to design, build and operate secure, scalable data pipelines for Government of Canada environments. You will work with Databricks, Lakeflow, Delta Lake, Spark, Python and SQL to deliver data products across modern data platforms.

The role requires 10+ years in IT, government clearance, and strong expertise in data engineering, ETL/ELT, and CI/CD practices. Remote, hybrid, or on-site options may apply.

Qualifikationen

• A minimum of 10 years of professional IT experience.
• A valid and active Government of Canada Reliability, Secret or Top Secret security clearance.
• Significant experience in data engineering, ETL/ELT or data integration.
• Hands-on Databricks experience.
• Strong Python, SQL, Apache Spark and/or PySpark experience.
• Experience designing enterprise pipelines and cloud-based data platforms, preferably Azure.
• Experience with data quality, troubleshooting and performance optimization.
• Experience with Git and modern software development practices.

Aufgaben

• Design, build and maintain enterprise data pipelines using Databricks.
• Build ingestion and transformation processes using Lakeflow, Lakeflow Connect, Jobs and Auto Loader.
• Develop batch and streaming solutions with Apache Spark and PySpark.
• Implement raw, cleaned and curated layers using Lakehouse and medallion patterns.
• Develop and optimize Delta Lake tables and enterprise data products.
• Connect Databricks with cloud, enterprise and on-premises sources.
• Modernize legacy ETL/ELT pipelines and data warehouse workloads.
• Implement data quality, validation, reconciliation, monitoring and Unity Catalog controls.
• Develop reusable engineering patterns and participate in CI/CD and automated deployments.
• Support testing, migration, production deployment, runbooks and client knowledge transfer.

Kenntnisse

Databricks

Python

SQL

Apache Spark

Git

Azure

Data engineering

Tools

Lakeflow

Delta Lake

Unity Catalog

Azure data services

Terraform

Databricks Asset Bundles

Azure DevOps or GitHub

Microsoft Purview

Jobbeschreibung

49 Solutions is proactively building a team of experienced, security-cleared professionals in preparation for anticipated Databricks initiatives within the Government of Canada. We are connecting with qualified candidates in advance so that we have a strong team ready to respond as relevant opportunities arise.

Please note that these opportunities are currently prospective and are not associated with a guaranteed placement or awarded contract. Candidates who join our network will be considered for relevant opportunities as they become available.

49 Solutions is seeking Senior Databricks Data Engineers to design, develop and operationalize secure, scalable data pipelines and data products for Government of Canada environments using Databricks, Lakeflow, Delta Lake, Spark, Python and SQL.

Responsibilities

• Design, build and maintain enterprise data pipelines using Databricks.
• Build ingestion and transformation processes using Lakeflow, Lakeflow Connect, Jobs and Auto Loader.
• Develop batch and streaming solutions with Apache Spark and PySpark.
• Implement raw, cleaned and curated layers using Lakehouse and medallion patterns.
• Develop and optimize Delta Lake tables and enterprise data products.
• Connect Databricks with cloud, enterprise and on-premises sources.
• Modernize legacy ETL/ELT pipelines and data warehouse workloads.
• Implement data quality, validation, reconciliation, monitoring and Unity Catalog controls.
• Develop reusable engineering patterns and participate in CI/CD and automated deployments.
• Support testing, migration, production deployment, runbooks and client knowledge transfer.

Mandatory Qualifications

• A minimum of 10 years of professional experience working in the IT industry or performing IT-related roles.
• A valid and active Government of Canada Reliability, Secret or Top Secret security clearance.
• Significant experience in data engineering, ETL/ELT or data integration.
• Demonstrated hands‑on Databricks experience.
• Strong Python, SQL, Apache Spark and/or PySpark experience.
• Experience designing enterprise pipelines and cloud-based data platforms, preferably Azure.
• Experience with data quality, troubleshooting and performance optimization.
• Experience with Git and modern software development practices.

Technical Skills

• Databricks
• Lakeflow
• Delta Lake
• Unity Catalog
• Apache Spark / PySpark
• Python
• SQLETL/ELT
• Batch and streaming processing
• Medallion architecture
• Azure data services
• Git and CI/CD

Assets

• Lakeflow Connect and Auto Loader
• Terraform or Databricks Asset Bundles
• Azure DevOps or GitHub
• Microsoft Purview
• On-premises source integration
• Migration from SAS, Synapse, DataStage, DB2, Oracle, SQL Server, Teradata or Netezza
• Government of Canada delivery experience
• Bilingualism in English and French

Certifications

Target Certification

• Databricks Certified Data Engineer Professional.

Candidates who do not currently hold the target certification may still be considered based on their experience. Candidates are not expected to obtain the certification immediately while 49 Solutions is building its prospective team. If a specific certification becomes required for an opportunity, candidates must be willing to obtain it at their own expense before being submitted for that opportunity.

Additional Asset Certifications

• Databricks EDW/ETL migration training
• Databricks Unity Catalog or data governance training
• Relevant Microsoft Azure data certifications
• Additional advanced Databricks certifications

Security Requirement

Candidates must hold a valid and active Government of Canada Reliability, Secret or Top Secret security clearance.

Location

Ottawa, Ontario, Canada. Engagements may be remote, hybrid, or performed at a Government of Canada location. Candidates located in Toronto or Montreal are also encouraged to apply, provided they are willing and able to travel to Ottawa for onsite meetings approximately once per month, as required.

Work arrangements will depend on the client, project and security requirements.

Use of Artificial Intelligence and Applicant Privacy

Use of Artificial Intelligence (AI tools): 49 Solutions may use approved, secure, and private enterprise AI tools, including Microsoft 365 Copilot, to support recruitment and proposal activities. Information processed using our approved enterprise AI tools is not used to train public AI models or underlying foundation models. Personal contact information (address, email address and phone numbers) will be removed before uploading into the enterprise AI environment. Personal Information shall not be entered into public or consumer AI systems, and our use of enterprise AI is subject to applicable privacy requirements, including PIPEDA, as well as organizational security, retention, and deletion policies.