Senior Data Engineer (Azure Databricks)

Akaasa Technologies; Houston (TX); On-site; USD 120,000 - 180,000; Part time

Job summary

Akaasa Technologies in Houston, TX seeks a Senior Data Engineer with deep hands-on expertise in Azure Databricks to design, develop, and manage enterprise-scale data platforms, using Delta Lake, PySpark, SQL and Scala to build scalable pipelines and modern data architectures.

You will migrate legacy workflows to Databricks-based solutions, implement data governance with Unity Catalog, and operate production pipelines with Databricks Workflows and Jobs, plus CI/CD automation through Azure DevOps.

Qualifications

• Strong hands-on experience with Azure Databricks engineering and development.
• Experience building data pipelines using PySpark, SQL, Scala, Delta Lake.
• Knowledge of Databricks Workflows/Jobs for production orchestration and Unity Catalog governance.

Responsibilities

• Design, develop, and optimize end-to-end data pipelines using Azure Databricks, Delta Lake, PySpark, SQL and Scala.
• Build and manage production workflows using Databricks Workflows and Jobs for orchestration.
• Lead migration of legacy data platforms to Databricks-based architectures.
• Convert legacy ETL workflows into scalable PySpark/SQL solutions.
• Implement data governance, security, and lineage using Unity Catalog.
• Develop and maintain CI/CD processes for Databricks deployments via Azure DevOps and YAML pipelines.

Skills

Azure Databricks

PySpark

SQL

Scala

Data Governance

Tools

Databricks Workflows

Databricks Jobs

Unity Catalog

Azure DevOps

CI/CD Pipelines

Job description

Senior Data Engineer (Azure Databricks)

Location: Houston, TX (Onsite)
Duration: 6 10 Months Contract
Interview: Virtual

Technical Skills: Azure Databricks | Delta Lake | PySpark | SQL | Scala | Databricks Workflows | Databricks Jobs | Unity Catalog | Azure DevOps | CI/CD | YAML Pipelines | Data Migration | ETL/ELT | Data Governance | Data Lineage | Alteryx Conversion | Azure Data Platform

Note:

• Must be local with DL copy
• LinkedIn with location and photo (before 2023)

Position Overview

Our client is seeking a Senior Data Engineer with deep hands-on expertise in Azure Databricks to design, develop, and manage enterprise-scale data platforms. The ideal candidate will have strong experience with Databricks Workflows, Delta Lake, PySpark, Unity Catalog, Azure DevOps, and CI/CD automation.

This role will focus on building code-driven data solutions, modernizing legacy data platforms, and migrating existing workflows into scalable Databricks-based architectures.

Required Skills & Experience

• Strong hands-on experience with Azure Databricks engineering and development.
• Expertise in building data pipelines using:
  • PySpark
  • SQL
  • Scala
  • Delta Lake
• Experience with Databricks Workflows and Jobs for production orchestration.
• Strong knowledge of Databricks Unity Catalog including governance, security, and lineage.
• Experience with Azure DevOps, CI/CD pipelines, YAML, and deployment automation.
• Experience migrating legacy data solutions into modern cloud data platforms.
• Strong understanding of enterprise data architecture and data engineering best practices.

Others Qualifications / Certifications

• Databricks certifications highly preferred:
  • Databricks Certified Data Engineer Associate
  • Databricks Certified Data Engineer Professional
• Microsoft certifications preferred:
  • Azure Data Engineer Associate (DP-203)
  • Azure Fundamentals (AZ-900) or equivalent Azure knowledge
• Experience converting Alteryx workflows into PySpark/SQL solutions is a major plus.
• Familiarity with legacy ETL modernization and reverse-engineering existing workflows.

Key Responsibilities

• Design, develop, and optimize end-to-end data pipelines using Azure Databricks, Delta Lake, PySpark, SQL, and Scala.
• Build and manage production workflows using Databricks Workflows and Jobs for pipeline orchestration and automation.
• Lead migration efforts from legacy data platforms into modern Databricks environments.
• Convert legacy ETL workflows (including Alteryx workflows) into scalable PySpark/SQL-based solutions.
• Implement data governance, security, access controls, schemas, and lineage using Databricks Unity Catalog.
• Develop and maintain CI/CD processes for Databricks deployments using Azure DevOps, YAML pipelines, and automation frameworks.
• Optimize data processing performance, reliability, and scalability across enterprise data platforms.