Sr Data Engineer

Golden Technology; Cincinnati (OH); On-site; USD 100,000 - 130,000; Full time

Job summary

A local contractor company is seeking an experienced Senior Databricks Engineer in Cincinnati, Ohio. This role requires deep hands-on expertise in designing and implementing large-scale data solutions on Azure Databricks, managing Delta Lake architecture, and optimizing performance. Responsibilities include developing data pipelines, implementing CI/CD practices, and troubleshooting production issues. Candidates should have 5+ years of relevant experience with strong skills in PySpark and Delta Lake.

Qualifications

• 5+ years of experience as a Data Engineer with Azure Databricks and PySpark.
• Solid understanding of Delta Lake, including Z-ordering and partitioning.

Responsibilities

• Design, develop, and maintain data pipelines in Azure Databricks using PySpark.
• Implement data integration frameworks and API-based ingestion.
• Analyze, design, and deliver enterprise data architecture solutions.
• Implement automation tools and CI/CD pipelines.
• Troubleshoot production failures and performance bottlenecks.
• Manage Unity Catalog and enforce data security.
• Administer Databricks clusters, optimizing costs.
• Collaborate with cross-functional teams on data strategy.
• Create and maintain architectural diagrams and documentation.
• Mentor junior engineers on Databricks and Azure data best practices.

Skills

Azure Databricks

PySpark

Data pipeline development

Delta Lake

CI/CD automation

Tools

Jenkins

Ansible

Terraform

Apigee

Kong

Job description

***This position is only for Local Nearby Contractor*****

We are seeking a Senior Databricks Engineer with deep hands-on experience designing and implementing large-scale data solutions on Azure Databricks. The ideal candidate has real-world experience building and troubleshooting production-grade data pipelines, optimizing Spark workloads, managing Delta Lake architecture, and implementing DevOps best practices using IaC and CI/CD automation.

Key Responsibilities

• Design, develop, and maintain data pipelines and ETL solutions in Azure Databricks using PySpark and Delta Lake.
• Implement data integration frameworks and API-based ingestion using tools like Apigee or Kong.
• Analyze, design, and deliver enterprise data architecture solutions focusing on scalability, performance, and governance.
• Implement automation tools and CI/CD pipelines using Jenkins, Ansible, or Terraform.
• Troubleshoot production failures and performance bottlenecks — fix partitioning, caching, shuffle, cluster sizing, and Z-ordering issues.
• Manage Unity Catalog, enforce data security (row/column-level access), and maintain data lineage.
• Administer Databricks clusters, jobs, and SQL warehouses, optimizing costs through auto-stop, job clusters, and Photon usage.
• Collaborate with cross-functional teams to drive data strategy and standards across domains.
• Create and maintain detailed architectural diagrams, interface specs, and data flow documentation.
• Mentor junior engineers on Databricks, Spark optimization, and Azure data best practices.

Required Skills & Experience

• 5+ years of experience as a Data Engineer with strong hands-on experience in Azure Databricks and PySpark.
• Solid understanding of Delta Lake, Z-ordering, partitioning, OPTIMIZE, and ACID transactions.