Senior Data Engineer - Azure Databricks & AI Pipelines

SharpAtoms; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

SharpAtoms seeks a Senior Data Engineer to design and operate scalable data pipelines using Databricks, PySpark, and Python on Azure. You will collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows.

You will also develop feature stores, real-time pipelines, and ensure data quality, governance, and security across the full data lifecycle, while driving reusable frameworks and CI/CD automation.

Qualifications

• 6+ years of experience in Data Engineering with Python/PySpark.
• 6+ years of experience in building ETL/ELT pipelines using Databricks.
• 6+ years of experience working in Agile environments.
• 5+ years of strong experience in SQL / PL-SQL.
• 4+ years of experience with Azure Databricks and Delta Lake architecture.
• 4+ years of hands-on experience with CI/CD (GitHub Actions, Azure DevOps).
• 3+ years of hands-on experience with Azure cloud services (ADF, ADLS, Databricks).
• 2+ years of experience with Databricks Delta Live Tables (DLT).
• 2+ years of experience with unit testing, validation, and pipeline testing frameworks.

Responsibilities

• Design and develop scalable data pipelines using Databricks, Apache Spark, and Python on Azure
• Build cloud-native solutions leveraging Azure Data Lake, Azure Data Factory, and Delta Lake
• Collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows
• Develop and maintain feature stores, model input pipelines, and real-time/streaming frameworks
• Ensure data quality, governance, and security across the full data lifecycle
• Build reusable frameworks, accelerators, and automation scripts to improve engineering efficiency
• Optimize performance, scalability, and reliability of data workflows and batch/streaming pipelines
• Participate in Agile development processes, including sprint planning, code reviews, and CI/CD pipelines
• Provide production support and on-call coverage, ensuring system stability and rapid issue resolution
• Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI

Skills

Python

PySpark

SQL

Agile methodologies

CI/CD

Azure Cloud

Data Engineering

Delta Lake

Databricks

Azure Data Factory

Education

Bachelor's degree in Computer Science, Engineering, or IT related field

Tools

Databricks

Delta Lake

Azure Data Factory

Azure Data Lake

GitHub Actions

Azure DevOps

DLT (Databricks Delta Live Tables)

Structured Streaming

Job description

SharpAtoms seeks a Senior Data Engineer to design and operate scalable data pipelines using Databricks, PySpark, and Python on Azure. You will collaborate with Data Science and AI teams to operationalize ML models and embed them into production workflows.

You will also develop feature stores, real-time pipelines, and ensure data quality, governance, and security across the full data lifecycle, while driving reusable frameworks and CI/CD automation.