Data Engineer - Databricks

Valiant Technologies LLC; New York (NY); In loco; USD 150.000 - 210.000; Tempo pieno

Descrizione del lavoro

Valiant Technologies LLC in New York, NY seeks a senior Data Engineer specializing in financial services data pipelines. You will own batch processing, integration of Bloomberg market feeds, and end-to-end ETL automation using Python, Spark, and SQL.

You will monitor, troubleshoot, and optimize data workflows across trading platforms, data ingestion, and reporting systems, working with operations, application support, and infrastructure teams.

Competenze

• 10+ years of data engineering or production support experience in financial services.
• Hands-on with Azure Databricks, Data Factory, and storage services.
• Strong Python and SQL programming skills.
• Experience with Bloomberg data feeds (BPIPE, TSIP, SFTP).
• Ability to monitor, troubleshoot, and optimize data pipelines.

Mansioni

• Manage and support batch processes and data pipelines in Azure Databricks and Data Factory.
• Integrate Bloomberg feeds into trading or analytics platforms.
• Monitor and resolve issues in trading applications and data ingestion.
• Develop and automate ETL pipelines using Python, Spark, and SQL.
• Ensure data quality, timeliness and completeness for downstream systems.
• Collaborate with operations, support, and infrastructure teams.
• Manage FTP/SFTP file transfers between internal systems and external vendors.

Conoscenze

Python

SQL

Azure Databricks

Azure Data Factory

Spark

ETL

Data quality

Strumenti

Azure Storage

Logic Apps

Fabric

Bloomberg feeds

Descrizione del lavoro

Job description

• Manage and support batch processes and data pipelines in Azure Databricks and Azure Data Factory.


• Integrate and process Bloomberg market data feeds and files into trading or analytics platforms.


• Monitor, troubleshoot, and resolve data and system issues related to trading applications and market data ingestion.


• Develop, automate, and optimize ETL pipelines using Python, Spark, and SQL.


• Manage FTP/SFTP file transfers between internal systems and external vendors.


• Ensure data quality, completeness, and timeliness for downstream trading and reporting systems.


• Collaborate with operations, application support, and infrastructure teams to resolve incidents and enhance data workflows.


• 10+ years of experience in data engineering or production support within financial services or trading environments.


• Hands-on experience with Azure Databricks, Azure Data Factory, and Azure Storage, Logic Apps, Fabric.


• Strong Python and SQL programming skills.


• Experience with Bloomberg data feeds (BPIPE, TSIP,SFTP).

VALIANT TECHNOLOGIES LLC is an equal opportunity employer inclusive of female, minority, disability and veterans, (M/F/D/V). Hiring, promotion, transfer, compensation, benefits, discipline, termination and all other employment decisions are made without regard to race, color, religion, sex, sexual orientation, gender identity, age, disability, national origin, citizenship/immigration status, veteran status or any other protected status. VALIANT TECHNOLOGIES LLC will not make any posting or employment decision that does not comply with applicable laws relating to labor and employment, equal opportunity, employment eligibility requirements or related matters. Nor will VALIANT TECHNOLOGIES LLC require in a posting or otherwise U.S. citizenship or lawful permanent residency in the U.S. as a condition of employment except as necessary to comply with law, regulation, executive order, or federal, state, or local government contract