Senior Databricks Engineer — AI-Driven Data Platform

SOMERSET STAFFING; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

SOMERSET STAFFING seeks an experienced Senior Data Engineer to design, build, and optimize scalable data platforms powering analytics, ML, and AI solutions in a cloud-native environment. You will work with Databricks, Spark, Delta Lake, Snowflake, Kafka, and AWS, contributing to GenAI, LLMs, and RAG-enabled engineering workflows.

You will lead data platform engineering, batch and streaming pipelines, and governance, while collaborating with data scientists and business stakeholders to deliver

Qualifications

• 8+ years of experience in software engineering, data engineering, or big data platform development.
• Strong experience designing and implementing enterprise-scale data pipelines.
• Hands-on expertise with Python, PySpark, Spark SQL, SQL,Kafka, Databricks, Delta Lake, Snowflake, Hive.
• Experience building data solutions on AWS cloud platforms.

Responsibilities

• Design, develop, and optimize scalable data platforms for analytics, ML, and AI-driven solutions.
• Build and maintain batch and real-time data pipelines and ingestion/processing frameworks.
• Implement Lakehouse architectures (Bronze/Silver/Gold) using Databricks, Delta Lake, and related patterns.
• Ensure data quality, governance, lineage, security, and compliance across ecosystems.
• Develop orchestration with Airflow and Databricks Workflows; enable CI/CD for data platforms.

Skills

Python

PySpark

Spark SQL

AWS

Databricks

Delta Lake

Kafka

Snowflake

Airflow

SQL

Education

Bachelor's or Master's degree in a related field

Tools

Databricks

Spark

Kafka

Snowflake

AWS

Delta Lake

Job description

SOMERSET STAFFING seeks an experienced Senior Data Engineer to design, build, and optimize scalable data platforms powering analytics, ML, and AI solutions in a cloud-native environment. You will work with Databricks, Spark, Delta Lake, Snowflake, Kafka, and AWS, contributing to GenAI, LLMs, and RAG-enabled engineering workflows.

You will lead data platform engineering, batch and streaming pipelines, and governance, while collaborating with data scientists and business stakeholders to deliver