Big Data Specialist

Hyrhub; New York (NY); On-site; USD 150,000 - 180,000; Full time

Job summary

Hyrhub is seeking a highly skilled Databricks SME to lead the architecture and implementation of enterprise-scale Big Data solutions on the Databricks Lakehouse Platform. The ideal candidate will have 12+ years of architecture experience and proven expertise in migrating data from legacy platforms like Alteryx.

This critical role is focused on driving data modernization initiatives and ensuring robust data governance across multiple systems. Proficiency in Python, SQL, PySpark, and strong knowledge of CI/CD practices is essential.

Qualifications

• 12+ years of experience in architecting Big Data solutions with Databricks.
• Proven track record in migrating pipelines from ETL tools like Alteryx.
• Expertise in building streaming data pipelines.

Responsibilities

• Architect enterprise Big Data solutions on the Databricks platform.
• Migrate data from legacy systems to Databricks.
• Implement CI/CD practices for data pipelines.

Skills

Databricks Lakehouse architecture

Data migration from Alteryx

Streaming pipeline design

CI/CD practices

Python

SQL

PySpark

Tools

Azure DevOps

GitHub Actions

Data quality tools

Job description

We are seeking a highly skilled and experienced Databricks SME to lead the architecture, design, and implementation of enterprise‑scale Big Data solutions on the Databricks Lakehouse Platform. The ideal candidate will have at least 12 years of hands‑on experience architecting big data solutions with Databricks, with proven expertise in migrating data and pipelines from legacy platforms like Alteryx, handling streaming data, and implementing CI/CD practices for data pipelines.

This role is critical for driving data modernization initiatives, ensuring robust data governance, and enabling scalable data ingestion and transformation across multiple source systems into the Databricks ecosystem.

Key Responsibilities

• Architect enterprise Big Data solutions on Databricks using Bronze/Silver/Gold (Medallion) layers with Delta Lake and Delta Live Tables
• Migrate data and pipelines from legacy platforms (Alteryx, Informatica, DataStage, Talend etc.) to Databricks, including source‑to‑target mapping and data validation
• Design streaming data pipelines using Structured Streaming, Auto Loader, and Delta Live Tables
• Implement CI/CD practices for data pipelines using Git, Azure DevOps, and GitHub Actions

Driving Innovation, Empowering Insights

• Develop data governance framework using Unity Catalog ABAC/RBAC policies and collaborate with stakeholders to translate requirements into scalable solutions
• Experience with Databricks Workflows, Jobs, and Databricks Asset Bundles for deployment automation
• Mentor junior team members and provide technical leadership across data engineering projects

Required Skills

• 12+ years architecting Big Data solutions with Databricks Lakehouse Platform
• Proven experience migrating pipelines from Alteryx or similar legacy ETL tools
• Hands‑on expertise with Delta Live Tables, Delta Lake, and Medallion architecture
• Strong experience building streaming pipelines with Structured Streaming and Auto Loader
• Demonstrated ability implementing CI/CD for data workloads
• Unity Catalog Data governance framework with ABAC / RBAC policies
• Proficient in Python, SQL, PySpark, and strong experience in performance tuning of large‑scale data
• Proven experience on at least one major cloud platform — Azure (preferred) or AWS — and its native data services

Preferred Skills

• Unity Catalog for data governance and lineage
• Data quality tools: Great Expectations, Informatica DQ, Collibra
• Data modeling tools: Erwin, dbt, ER/Studio
• AI/ML experience with GenAI, LLMs, MLOps

Certifications (Preferred)

• Databricks Certified Data Engineer Associate/Professional