Databricks Engineer – AWS Lakehouse & Real-Time Data

SMBC; New York (NY); Hybrid; USD 73,000 - 98,000; Full time

Benefits offered by this job

Hybrid work model

Competitive benefits

Job summary

SMBC seeks a Databricks Engineer with AWS expertise to design, build, and maintain an enterprise data Lakehouse infrastructure in the US. The role focuses on high-performance data pipelines, real-time reporting, and analytics environments, integrating Databricks with core AWS services to support real-time financial data consumption.

You will collaborate with data scientists, risk managers, and analytics teams to deliver clean, business-ready data views for reporting and AI modeling, while

Qualifications

• Experience designing and building Databricks-based data lakes on AWS.
• Strong Python (PySpark) and SQL for data transformations.
• Hands-on with Spark internals, Delta Lake, and streaming data concepts.
• Experience with CI/CD tooling and automated deployments.

Responsibilities

• Design and implement Databricks Lakehouse data pipelines on AWS.
• Build pipelines using Medallion Architecture (Bronze/Silver/Gold).
• Automate data workflows with Databricks Jobs and Delta Live Tables.
• Standardize deployment of Databricks assets across environments.
• Integrate Databricks with S3, Glue, and IAM for secure access.
• Tune Spark clusters and Delta Lake for performance and cost.
• Implement data governance with Unity Catalog and auditing.
• Collaborate with data scientists, risk managers, and analytics teams.

Skills

Databricks

Python (PySpark)

SQL

Apache Spark

Delta Lake

AWS Cloud

Kafka/Streaming

Data Lakehouse

Tools

Terraform

Databricks Asset Bundles

GitHub Actions / GitLab CI

Delta Live Tables

Amazon S3

AWS Glue

Unity Catalog

Job description

SMBC seeks a Databricks Engineer with AWS expertise to design, build, and maintain an enterprise data Lakehouse infrastructure in the US. The role focuses on high-performance data pipelines, real-time reporting, and analytics environments, integrating Databricks with core AWS services to support real-time financial data consumption.

You will collaborate with data scientists, risk managers, and analytics teams to deliver clean, business-ready data views for reporting and AI modeling, while