Senior Databricks & AI Data Platform Engineer

Expert Technology Services; New York (NY); On-site; USD 140,000 - 180,000; Full time

Job summary

Expert Technology Services is seeking a Senior Data Engineer in New York to design, build, and optimize scalable data pipelines for enterprise analytics, ML, and AI-driven solutions.

You will implement Lakehouse architectures with Databricks, Delta Lake, and pattern-driven Bronze-Silver-Gold data processing, while ensuring data governance and security across complex ecosystems. Strong AWS, Kafka, and CI/CD experience are required.

Qualifications

• 8+ years in data/software/big data engineering roles.
• Deep expertise with Python, PySpark, Spark SQL, Databricks, Delta Lake, Snowflake, AWS, and Kafka.
• Strong experience with distributed computing, data modeling, and large-scale data processing.
• Hands-on experience with CI/CD, orchestration, and automation frameworks.
• Knowledge of data governance, metadata management, and data quality frameworks.
• Experience with Generative AI, LLMs, RAG, and developer productivity tooling.
• AWS certifications and exposure to supply chain, retail, healthcare, or manufacturing data domains are a plus.

Responsibilities

• Design, build, and optimize scalable batch and real-time data pipelines for analytics, ML, and AI-driven solutions.
• Develop and maintain data ingestion, transformation, and processing frameworks for various data types (structured, semi-structured, unstructured).
• Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion design patterns (Bronze, Silver, Gold).
• Ensure data quality, governance, lineage, security, and compliance across complex data ecosystems.
• Develop distributed data processing and streaming solutions with PySpark, Spark SQL, Kafka, and AWS Kinesis.
• Architect and deploy cloud-based data solutions on AWS (S3, EMR, EC2, Athena, Redshift, Lambda).
• Support migration from Hadoop/EMR to modern cloud-native platforms.
• Build and automate orchestration, scheduling, and CI/CD pipelines using Airflow, Databricks Workflows.
• Collaborate with architects, data scientists, analysts, and stakeholders to deliver enterprise-grade solutions.
• Design and implement Generative AI-powered and AI-assisted solutions to enhance engineering productivity and platform intelligence.

Skills

Python

PySpark

Spark SQL

Databricks

Delta Lake

Snowflake

AWS

Kafka

CI/CD

Data governance

Generative AI

LLMs

Tools

Airflow

Databricks Workflows

Kinesis

Lambda

Job description

Expert Technology Services is seeking a Senior Data Engineer in New York to design, build, and optimize scalable data pipelines for enterprise analytics, ML, and AI-driven solutions.

You will implement Lakehouse architectures with Databricks, Delta Lake, and pattern-driven Bronze-Silver-Gold data processing, while ensuring data governance and security across complex ecosystems. Strong AWS, Kafka, and CI/CD experience are required.