Sr. Databricks Engineer - Hybrid NYC

Expert Technology Services; New York (NY); On-site; USD 140,000 - 180,000; Full time

Job summary

Expert Technology Services is seeking a Senior Data Engineer in New York to design, build, and optimize scalable data pipelines for enterprise analytics, ML, and AI-driven solutions.

You will implement Lakehouse architectures with Databricks, Delta Lake, and pattern-driven Bronze-Silver-Gold data processing, while ensuring data governance and security across complex ecosystems. Strong AWS, Kafka, and CI/CD experience are required.

Qualifications

• 8+ years in data/software/big data engineering roles.
• Deep expertise with Python, PySpark, Spark SQL, Databricks, Delta Lake, Snowflake, AWS, and Kafka.
• Strong experience with distributed computing, data modeling, and large-scale data processing.
• Hands-on experience with CI/CD, orchestration, and automation frameworks.
• Knowledge of data governance, metadata management, and data quality frameworks.
• Experience with Generative AI, LLMs, RAG, and developer productivity tooling.
• AWS certifications and exposure to supply chain, retail, healthcare, or manufacturing data domains are a plus.

Responsibilities

• Design, build, and optimize scalable batch and real-time data pipelines for analytics, ML, and AI-driven solutions.
• Develop and maintain data ingestion, transformation, and processing frameworks for various data types (structured, semi-structured, unstructured).
• Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion design patterns (Bronze, Silver, Gold).
• Ensure data quality, governance, lineage, security, and compliance across complex data ecosystems.
• Develop distributed data processing and streaming solutions with PySpark, Spark SQL, Kafka, and AWS Kinesis.
• Architect and deploy cloud-based data solutions on AWS (S3, EMR, EC2, Athena, Redshift, Lambda).
• Support migration from Hadoop/EMR to modern cloud-native platforms.
• Build and automate orchestration, scheduling, and CI/CD pipelines using Airflow, Databricks Workflows.
• Collaborate with architects, data scientists, analysts, and stakeholders to deliver enterprise-grade solutions.
• Design and implement Generative AI-powered and AI-assisted solutions to enhance engineering productivity and platform intelligence.

Skills

Python

PySpark

Spark SQL

Databricks

Delta Lake

Snowflake

AWS

Kafka

CI/CD

Data governance

Generative AI

LLMs

Tools

Airflow

Databricks Workflows

Kinesis

Lambda

Job description

Job Summary (Senior Data Engineer)

• - Design, build, and optimize scalable batch and real-time data pipelines for enterprise analytics, machine learning, and AI-driven solutions.
• - Develop and maintain data ingestion, transformation, and processing frameworks for various data types (structured, semi-structured, unstructured).
• - Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion design patterns (Bronze, Silver, Gold).
• - Ensure data quality, governance, lineage, security, and compliance across complex data ecosystems.
• - Develop distributed data processing and streaming solutions with PySpark, Spark SQL, Kafka, and AWS Kinesis.
• - Architect and deploy cloud-based data solutions on AWS, leveraging services like S3, EMR, EC2, Athena, Redshift, Lambda, and more.
• - Support migration from traditional Hadoop and EMR environments to modern cloud-native platforms.
• - Build and automate orchestration, scheduling, and CI/CD pipelines using Airflow, Databricks Workflows, and related frameworks.
• - Collaborate with architects, data scientists, analysts, and business stakeholders to deliver enterprise-grade solutions.
• - Design and implement Generative AI-powered and AI-assisted solutions to enhance engineering productivity and platform intelligence.
• - Evaluate and integrate emerging AI technologies including LLMs, RAG, Vector Databases, and Model Context Protocol (MCP).
• - Maintain high standards for system reliability, scalability, performance, and cost optimization.
• - Contribute to the modernization and innovation of data platforms, driving adoption of cloud-native and AI-driven architectures.

Ideal Candidate Will Have:

• - 8+ years in data/software/big data engineering roles.
• - Deep expertise with Python, PySpark, Spark SQL, Databricks, Delta Lake, Snowflake, AWS, and Kafka.
• - Strong experience with distributed computing, data modeling, and large-scale data processing.
• - Hands-on experience with CI/CD, orchestration, and automation frameworks.
• - Knowledge of data governance, metadata management, and data quality frameworks.
• - Experience with Generative AI, LLMs, RAG, and developer productivity tooling.
• - AWS certifications and exposure to supply chain, retail, healthcare, or manufacturing data domains are a plus.