Senior Databricks Specialist / Data Architect

Malomatia

Doha

On-site

QAR 300,000 - 420,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Malomatia in Doha is seeking a hands-on senior Databricks specialist to independently design, develop, troubleshoot, optimize, and govern Databricks solutions in production enterprise environments.

Responsibilities include designing enterprise-scale data solutions, data architecture, ingestion pipelines, ETL/ELT with Spark, Medallion architecture, Delta Lake optimization, data modeling, and data governance. Strong expertise in Azure data services and CI/CD is required.

Qualifications

  • Minimum 8+ years of experience in data engineering/data architecture.
  • Minimum 4+ years of hands-on experience with Azure Databricks.
  • Strong hands-on experience with Apache Spark and PySpark.
  • Strong experience with Delta Lake.
  • Strong experience designing enterprise data platforms and data pipelines.
  • Strong knowledge of Databricks Medallion Architecture.
  • Strong experience with Unity Catalog, data governance, access control, and data lineage.
  • Strong SQL skills and experience with large-scale data processing.
  • Strong experience with Azure data services.
  • Experience with Azure Data Factory / Azure Data Lake Storage Gen2.
  • Experience integrating Databricks with APIs, enterprise apps, CRM systems, and other data sources.
  • Experience with data modeling and dimensional modeling.
  • Strong understanding of data quality and governance.
  • Strong understanding of performance optimization and Databricks cost optimization.
  • Experience with CI/CD, Git, Azure DevOps, and deployment automation.
  • Experience working with large-volume enterprise datasets.
  • Ability to troubleshoot complex production data issues.

Responsibilities

  • Design and implement enterprise-scale data solutions using Azure Databricks.
  • Define and maintain Databricks data architecture and platform standards.
  • Design data ingestion and integration pipelines from multiple source systems.
  • Develop scalable ETL/ELT pipelines using Databricks and Apache Spark.
  • Design and implement Medallion Architecture (Bronze, Silver, Gold) where appropriate.
  • Work with Delta Lake and optimize Delta tables and data pipelines.
  • Design data models and transformation processes for analytical and operational reporting.
  • Integrate Databricks with Microsoft Dynamics 365 / Dataverse and other enterprise applications.
  • Develop and optimize batch and near-real-time data processing pipelines.
  • Implement data quality, validation, monitoring, lineage, and governance processes.
  • Optimize Spark jobs, clusters, SQL queries, Delta tables, and data pipelines for performance and cost.
  • Implement appropriate data security and access controls.
  • Work with Unity Catalog for data governance, access control, cataloging, and lineage.
  • Support Power BI and other analytical/reporting platforms through optimized data models and data services.
  • Troubleshoot complex data engineering and performance issues.
  • Establish development, testing, and production standards for Databricks.
  • Support CI/CD and DevOps practices for Databricks solutions.
  • Conduct technical reviews and mentor data engineers.
  • Ensure data platform solutions meet security, availability, scalability, and disaster recovery requirements.

Job description

Job Description

Hands-on senior Databricks specialist with proven experience in production enterprise environments. We are specifically looking for someone who can independently design, develop, troubleshoot, optimize, and govern Databricks solutions, rather than someone whose experience is limited to basic Databricks development.

Responsibilities
  • Design and implement enterprise-scale data solutions using Azure Databricks.
  • Define and maintain Databricks data architecture and platform standards.
  • Design data ingestion and integration pipelines from multiple source systems.
  • Develop scalable ETL/ELT pipelines using Databricks and Apache Spark.
  • Design and implement Medallion Architecture (Bronze, Silver, Gold) where appropriate.
  • Work with Delta Lake and optimize Delta tables and data pipelines.
  • Design data models and transformation processes for analytical and operational reporting.
  • Integrate Databricks with Microsoft Dynamics 365 / Dataverse and other enterprise applications.
  • Develop and optimize batch and near-real-time data processing pipelines.
  • Implement data quality, validation, monitoring, lineage, and governance processes.
  • Optimize Spark jobs, clusters, SQL queries, Delta tables, and data pipelines for performance and cost.
  • Implement appropriate data security and access controls.
  • Work with Unity Catalog for data governance, access control, cataloging, and lineage.
  • Support Power BI and other analytical/reporting platforms through optimized data models and data services.
  • Troubleshoot complex data engineering and performance issues.
  • Establish development, testing, and production standards for Databricks.
  • Support CI/CD and DevOps practices for Databricks solutions.
  • Conduct technical reviews and mentor data engineers.
  • Ensure data platform solutions meet security, availability, scalability, and disaster recovery requirements.
Qualifications
  • Minimum 8+ years of experience in data engineering/data architecture.
  • Minimum 4+ years of strong hands-on experience with Azure Databricks.
  • Strong hands-on experience with Apache Spark and PySpark.
  • Strong experience with Delta Lake.
  • Strong experience designing enterprise data platforms and data pipelines.
  • Strong knowledge of Databricks Medallion Architecture.
  • Strong experience with Unity Catalog, data governance, access control, and data lineage.
  • Strong SQL skills and experience with large-scale data processing.
  • Strong experience with Azure data services.
  • Experience with Azure Data Factory / Azure Data Lake Storage Gen2.
  • Experience integrating Databricks with APIs, enterprise applications, CRM systems, and other data sources.
  • Experience with data modeling and dimensional modeling.
  • Strong understanding of data quality and data governance.
  • Strong understanding of performance optimization and Databricks cost optimization.
  • Experience with CI/CD, Git, Azure DevOps, and deployment automation.
  • Experience working with large-volume enterprise datasets.
  • Ability to troubleshoot complex production data issues.
Preferred Certifications
  • Databricks Certified Data Engineer Professional
  • Databricks Certified Data Engineer Associate
  • Microsoft Certified: Azure Data Engineer Associate
  • Other relevant Databricks/Azure certifications
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks Specialist / Data Architect
Senior Databricks Specialist / Data Architect

Employment • Doha

On-site
QAR 300,000 - 520,000
Azure Databricks Architect & Data Platform Lead
Azure Databricks Architect & Data Platform Lead

Employment • Doha

On-site
QAR 300,000 - 520,000
Senior Azure Databricks Architect
Senior Azure Databricks Architect

Malomatia • Doha

On-site
QAR 300,000 - 420,000
Senior Databricks Specialist/Data Architect | Marc Ellis | Doha, Qatar
Senior Databricks Specialist/Data Architect | Marc Ellis | Doha, Qatar

Marc Ellis • Doha

On-site
QAR 437,000 - 619,000
Senior Databricks Architect (Azure) – Delta Lake & Unity
Senior Databricks Architect (Azure) – Delta Lake & Unity

Marc Ellis • Doha

On-site
QAR 437,000 - 619,000
Senior Data Engineer
Senior Data Engineer

Management Solutions International • Doha

On-site
QAR 300,000 - 540,000
Sr. Data Architect Specialist
Sr. Data Architect Specialist

Confidential • Doha

On-site
QAR 360,000 - 540,000
Senior Data Engineer
Senior Data Engineer

Malomatia • Doha

On-site
QAR 250,000 - 400,000
Technical Lead – Microsoft Dynamics 365
Technical Lead – Microsoft Dynamics 365

High Tech Genesis • Doha

On-site
QAR 320,000 - 420,000
Data Engineer / Data Architect
Data Engineer / Data Architect

Power International Holding • Doha

On-site
QAR 250,000 - 420,000
Top locations