Data Engineer - Databricks

Cloud Analytics Technologies, LLC; New York (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

A financial data services company in New York, NY is seeking an experienced data engineer to manage batch processes and data pipelines in Azure. The ideal candidate will have over 10 years of experience in data engineering, particularly in financial services, along with hands-on expertise in Azure tools and strong programming skills in Python and SQL. This role offers an opportunity to work closely with data and trading applications, ensuring data quality and enhancing workflows.

Qualifications

• 10+ years of experience in data engineering or production support within financial services or trading environments.
• Hands-on experience with Azure Databricks and Azure Data Factory.
• Strong Python and SQL programming skills.

Responsibilities

• Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.
• Integrate and process Bloomberg market data feeds into trading platforms.
• Monitor and resolve data and system issues related to trading applications.

Skills

Data engineering

Production support

Azure Databricks

Python

SQL

Tools

Azure Data Factory

Bloomberg data feeds

Job description

Responsibilities

• Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.
• Integrate and process Bloomberg market data feeds and files into trading or analytics platforms.
• Monitor, troubleshoot, and resolve data and system issues related to trading applications and market data ingestion.
• Develop, automate, and optimize ETL pipelines using Python, Spark, and SQL.
• Manage FTP/SFTP file transfers between internal systems and external vendors.
• Ensure data quality, completeness, and timeliness for downstream trading and reporting systems.
• Collaborate with operations, application support, and infrastructure teams to resolve incidents and enhance data workflows.

Qualifications

• 10+ years of experience in data engineering or production support within financial services or trading environments.
• Hands-on experience with Azure Databricks, Azure Data Factory, and Azure Storage, Logic Apps, Fabric.
• Strong Python and SQL programming skills.
• Experience with Bloomberg data feeds (BPIPE, TSIP, SFTP).

Equal Opportunity Employer: CLOUD ANALYTICS TECHNOLOGIES LLC is an equal opportunity employer inclusive of female, minority, disability and veterans (M/F/D/V). Hiring, promotion, transfer, compensation, benefits, discipline, termination and all other employment decisions are made without regard to race, color, religion, sex, sexual orientation, gender identity, age, disability, national origin, citizenship/immigration status, veteran status or any other protected status. CLOUD ANALYTICS TECHNOLOGIES LLC will comply with applicable laws relating to labor and employment and equal opportunity. No requirement for U.S. citizenship or permanent residency beyond what is necessary to comply with law and government contracts.