Data Engineer - Databricks

MACHINE LEARNING TECHNOLOGIES LLC; New York (NY); On-site; USD 150,000 - 210,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC in New York seeks a seasoned Data Engineer to design and maintain data pipelines in Azure Databricks and Azure Data Factory for trading and analytics. You will ingest Bloomberg market data, ensure data quality, and collaborate with operations and infrastructure teams to resolve issues while advancing data workflows.

A strong background in Python and SQL is required, with 10+ years in financial services data engineering.

Qualifications

• 10+ years of experience in data engineering or production support within financial services or trading environments.
• Hands-on experience with Azure Databricks, Azure Data Factory, and Azure Storage, Logic Apps, Fabric.
• Strong Python and SQL programming skills.
• Experience with Bloomberg data feeds (BPIPE, TSIP,SFTP).
• Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.

Responsibilities

• Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.
• Integrate and process Bloomberg market data feeds and files into trading or analytics platforms.
• Monitor, troubleshoot, and resolve data and system issues related to trading applications and market data ingestion.
• Develop, automate, and optimize ETL pipelines using Python, Spark, and SQL.
• Manage FTP/SFTP file transfers between internal systems and external vendors.
• Ensure data quality, completeness, and timeliness for downstream trading and reporting systems.
• Collaborate with operations, application support, and infrastructure teams to resolve incidents and enhance data workflows.

Skills

Python programming

SQL programming

Data pipelines

Batch processing

Tools

Azure Databricks

Azure Data Factory

Azure Storage

Logic Apps

Fabric

Bloomberg data feeds

Job description

Job Overview

Job ID: J52299

Job Title: Data Engineer - Databricks

Location: New york, NY

Duration: 17 Months + Extension

Hourly Rate: Depending on Experience (DOE)

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, L2-EAD, GC-EAD

Client: To Be Discussed Later

Employment Type: W-2, 1099, C2C

Similar Jobs

• Data Scientist - Lathrop, CA
• Senior Data Scientist (Python, AI ML) - Alpharetta, GA
• Data Scientist Lead - Charlotte, NC
• Data Scientist - Raleigh, NC
• Data Scientist - Nashville, TN

Job Description

• Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.
• Integrate and process Bloomberg market data feeds and files into trading or analytics platforms.
• Monitor, troubleshoot, and resolve data and system issues related to trading applications and market data ingestion.
• Develop, automate, and optimize ETL pipelines using Python, Spark, and SQL.
• Manage FTP/SFTP file transfers between internal systems and external vendors.
• Ensure data quality, completeness, and timeliness for downstream trading and reporting systems.
• Collaborate with operations, application support, and infrastructure teams to resolve incidents and enhance data workflows.
• 10+ years of experience in data engineering or production support within financial services or trading environments.
• Hands-on experience with Azure Databricks, Azure Data Factory, and Azure Storage, Logic Apps, Fabric.
• Strong Python and SQL programming skills.
• Experience with Bloomberg data feeds (BPIPE, TSIP,SFTP).

Equal Opportunity Employer

MACHINE LEARNING TECHNOLOGIES LLC is an equal opportunity employer inclusive of female, minority, disability and veterans, (M/F/D/V). Hiring, promotion, transfer, compensation, benefits, discipline, termination and all other employment decisions are made without regard to race, color, religion, sex, sexual orientation, gender identity, age, disability, national origin, citizenship/immigration status, veteran status or any other protected status. MACHINE LEARNING TECHNOLOGIES LLC will not make any posting or employment decision that does not comply with applicable laws relating to labor and employment, equal opportunity, employment eligibility requirements or related matters. Nor will MACHINE LEARNING TECHNOLOGIES LLC require in a posting or otherwise U.S. citizenship or lawful permanent residency in the U.S. as a condition of employment except as necessary to comply with law, regulation, executive order, or federal, state, or local government contract