Databricks Engineer – AWS Lakehouse & Real-Time Data

SMBC; New York (NY); Híbrido; USD 73.000 - 98.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Hybrid work model

Competitive benefits

Descripción de la vacante

SMBC seeks a Databricks Engineer with AWS expertise to design, build, and maintain an enterprise data Lakehouse infrastructure in the US. The role focuses on high-performance data pipelines, real-time reporting, and analytics environments, integrating Databricks with core AWS services to support real-time financial data consumption.

You will collaborate with data scientists, risk managers, and analytics teams to deliver clean, business-ready data views for reporting and AI modeling, while

Formación

• Experience designing and building Databricks-based data lakes on AWS.
• Strong Python (PySpark) and SQL for data transformations.
• Hands-on with Spark internals, Delta Lake, and streaming data concepts.
• Experience with CI/CD tooling and automated deployments.

Responsabilidades

• Design and implement Databricks Lakehouse data pipelines on AWS.
• Build pipelines using Medallion Architecture (Bronze/Silver/Gold).
• Automate data workflows with Databricks Jobs and Delta Live Tables.
• Standardize deployment of Databricks assets across environments.
• Integrate Databricks with S3, Glue, and IAM for secure access.
• Tune Spark clusters and Delta Lake for performance and cost.
• Implement data governance with Unity Catalog and auditing.
• Collaborate with data scientists, risk managers, and analytics teams.

Conocimientos

Databricks

Python (PySpark)

SQL

Apache Spark

Delta Lake

AWS Cloud

Kafka/Streaming

Data Lakehouse

Herramientas

Terraform

Databricks Asset Bundles

GitHub Actions / GitLab CI

Delta Live Tables

Amazon S3

AWS Glue

Unity Catalog

Descripción del empleo

SMBC seeks a Databricks Engineer with AWS expertise to design, build, and maintain an enterprise data Lakehouse infrastructure in the US. The role focuses on high-performance data pipelines, real-time reporting, and analytics environments, integrating Databricks with core AWS services to support real-time financial data consumption.

You will collaborate with data scientists, risk managers, and analytics teams to deliver clean, business-ready data views for reporting and AI modeling, while