Senior Databricks Engineer for AI-Driven Data Platforms

hyrusinc; New York (NY); On-site; USD 130,000 - 190,000; Full time

Job summary

hyrusinc seeks an experienced Senior Data Engineer to design and optimize scalable data platforms for enterprise analytics and AI-driven solutions. You will build real-time and batch pipelines, implement lakehouse architectures, and ensure data governance across AWS services including S3, EMR, Redshift, and Lambda.

The role requires 8+ years in data engineering, strong PySpark/Spark SQL skills, and hands-on experience with Databricks, Delta Lake, Snowflake, Kafka, and Hive.

Qualifications

• 8+ years of experience in software engineering, data engineering, or big data platform development.
• Strong experience designing and implementing enterprise-scale data pipelines.
• Hands-on expertise with Python, PySpark, Spark SQL, SQL, Kafka, Databricks, Delta Lake, Snowflake, Hive.
• Experience building data solutions on AWS cloud platforms.
• Strong understanding of distributed computing, data modeling, and large-scale data processing.
• Experience with Git-based development workflows and CI/CD practices.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Familiarity with data governance, metadata management, and data quality frameworks (preferred).

Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines.
• Develop ingestion, transformation, and processing frameworks for structured, semi-structured, and unstructured datasets.
• Implement modern Lakehouse architectures utilizing Databricks, Delta Lake, and Medallion design patterns.
• Build data solutions that support analytics, reporting, and machine learning initiatives.
• Ensure data quality, governance, lineage, security, and compliance across ecosystems.
• Develop distributed data processing apps using PySpark and Spark SQL.
• Create streaming pipelines with Kafka, Spark Structured Streaming, and AWS Kinesis.
• Architect and implement cloud-based data solutions on AWS (S3, EMR, Redshift, etc.).
• Collaborate with architects, analysts, data scientists, and stakeholders to deliver enterprise-grade solutions.
• Evaluate AI technologies and opportunities for adoption within data engineering processes.

Skills

Analytical thinking

Problem solving

Team collaboration

Excellent communication

Education

Bachelor's or Master's degree in Computer Engineering, Computer Science, Information Systems, or related field

Tools

Python

PySpark

Spark SQL

SQL

Kafka

Databricks

Delta Lake

Snowflake

Hive

Git

CI/CD

Job description

hyrusinc seeks an experienced Senior Data Engineer to design and optimize scalable data platforms for enterprise analytics and AI-driven solutions. You will build real-time and batch pipelines, implement lakehouse architectures, and ensure data governance across AWS services including S3, EMR, Redshift, and Lambda.

The role requires 8+ years in data engineering, strong PySpark/Spark SQL skills, and hands-on experience with Databricks, Delta Lake, Snowflake, Kafka, and Hive.