AI Databricks Engineer - New York

Caspian One; New York (NY); Hybrid; USD 120,000 - 170,000; Full time

Job summary

Caspian One invites an AI Databricks Engineer to join the buy-side asset management team’s engineering function in New York. The role focuses on building scalable data solutions in Databricks and Azure, partnering with US stakeholders to translate business needs into technical outcomes, and supporting reporting and AI initiatives.

This contract-to-hire position operates in a hybrid New York environment, offering hands-on data platform ownership, governance, and continuous improvement within a

Qualifications

• Strong Data Engineering experience with Databricks, Azure, Python/PySpark and SQL, including hands-on ownership of Databricks environments, data pipelines, governance and CDC.

Responsibilities

• Build and maintain scalable data solutions and pipelines within Databricks and Azure.
• Partner with US business stakeholders to understand requirements and deliver practical data-driven solutions.
• Translate business needs into technical outcomes for the engineering team.
• Support reporting, analytics and AI-related initiatives across the organisation.
• Contribute to data platform development, governance, optimisation and best practices.
• Work closely with a small, collaborative engineering team to solve complex data challenges and drive continuous improvements.

Skills

Databricks

Azure

Python/PySpark

SQL

CDC

GitHub Actions

CI/CD

Data pipelines

Data governance

Tools

Tableau

Sigma

Job description

AI Databricks Engineer - Buy-Side Asset Management | Hybrid - New York (4 Days Per Week) - 12-Month - Contract to Hire Opportunity

We're partnering with a leading buy-side asset manager to hire a Data Engineer into their Engineering team. This opportunity offers the chance to become a key technical individual to the firm's growing US business.

The role sits within a small, highly collaborative engineering function and would suit someone who enjoys combining hands-on engineering with direct business engagement.

Key Responsibilities:

• Build and maintain scalable data solutions and pipelines within Databricks and Azure.


• Partner with US business stakeholders to understand requirements and deliver practical data-driven solutions.


• Act as a bridge between the business and engineering teams, translating business needs into technical outcomes.


• Support reporting, analytics and AI-related initiatives across the organisation.


• Contribute to data platform development, governance, optimisation and best practices.


• Work closely with a small, collaborative engineering team to solve complex data challenges and drive continuous improvements.

Required Experience:

• Strong Data Engineering experience with Databricks, Azure, Python/PySpark and SQL, including hands-on ownership of Databricks environments, data pipelines, governance, security and Change Data Capture (CDC).


• Experience building and supporting cloud-based data platforms, integrating data from external APIs, deploying production-ready data assets, and independently diagnosing and resolving pipeline and data quality issues.


• Strong understanding of modern engineering practices, including Git-based workflows, CI/CD (e.g. GitHub Actions), orchestration tooling, monitoring, validation frameworks and performance optimisation.


• Experience designing scalable data models and working with complex datasets. Exposure to financial datasets such as trade, reference, position or time-series data would be highly beneficial.


• Comfortable working in a fast-paced environment with evolving requirements, partnering directly with stakeholders to translate business needs into technical solutions and independently support business users.


• Experience with reporting tools such as Tableau or Sigma, platform engineering, AI-assisted development tools (Copilot, Cursor, Claude Code), or investment management environments would be highly advantageous.


• Strong communication skills with the ability to explain technical concepts to non-technical audiences.

Why Apply?

• High visibility role supporting the US business.


• Direct exposure to stakeholders and decision makers.


• Opportunity to work closely with engineering leadership.


• Mix of technical ownership and business engagement.


• Modern cloud and data technology stack.


• Small team environment with significant impact and autonomy.