Hybrid Data Engineer Associate: Azure Databricks Pipelines

SMBC; Charlotte (NC); Hybrid; USD 104,000 - 170,000; Full time

Benefits offered by this job

Hybrid work model

Accommodations for disabilities

Job summary

SMBC is seeking an Associate Data Engineer to join the Trade Data Repository team within Market Surveillance Technology. You will help develop, monitor, and modernize a critical data platform for ingested trade and order data, currently on Azure SQL Server and Blob Storage and migrating to Azure Databricks Lakehouse.

You will work with senior engineers, analysts, and governance teams to build reliable data pipelines supporting regulatory and compliance objectives and ensure data is accurate,

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, Data Analytics, or related discipline.
• 2-5 years of experience in Data Engineering, ETL Development, or Data Warehousing.
• Experience with financial services, capital markets, risk, compliance, or surveillance data preferred.

Responsibilities

• Develop and maintain SQL-based ingestion, transformation, and reconciliation processes within the TDR platform.
• Design and optimize complex SQL queries, stored procedures, views, functions, and ETL processes.
• Build and support Azure Data Factory (ADF) data pipelines for ingestion of structured and semi-structured data.
• Develop Databricks notebooks using PySpark, Spark SQL, and Python.
• Participate in migration of TDR workloads from Azure SQL Server to Azure Databricks Lakehouse.
• Support implementation of Bronze (Pre-Stage), Silver (Stage), and Gold (Outbound) data layers.
• Develop integrations involving: Build reusable ingestion and transformation frameworks.
• Support batch processing, incremental loads, and historical data processing.
• Implement metadata-driven pipeline designs and parameterized frameworks.

Skills

SQL

PySpark

Python

Data modeling

Data validation

Git

CI/CD

Delta Lake

Education

Bachelor's degree in CS/IS

Tools

Azure Databricks

Azure Data Factory

Azure SQL Server

Azure DevOps

Unity Catalog

Job description

SMBC is seeking an Associate Data Engineer to join the Trade Data Repository team within Market Surveillance Technology. You will help develop, monitor, and modernize a critical data platform for ingested trade and order data, currently on Azure SQL Server and Blob Storage and migrating to Azure Databricks Lakehouse.

You will work with senior engineers, analysts, and governance teams to build reliable data pipelines supporting regulatory and compliance objectives and ensure data is accurate,