Databricks Architect

Newt Global; Irving (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Newt Global seeks a highly skilled Databricks Engineer with AWS expertise to design, develop, and optimize enterprise-scale data platforms, pipelines, and analytics solutions. You will collaborate with data architects, application teams, business stakeholders, and platform engineering to deliver scalable, secure, and high-performance data solutions.

You will design and maintain scalable pipelines using Databricks, PySpark, and Delta Lake, build lakehouse solutions on AWS, and implement CI/CD

Qualifications

• Proficient in designing scalable data pipelines with Databricks and PySpark.
• Strong knowledge of data lakehouse architectures on AWS.
• Experience with data modeling, data quality, and CI/CD for data platforms.
• Familiarity with batch and real-time data processing and performance tuning.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Spark SQL, and Delta Lake.
• Build and support enterprise data lake and Lakehouse solutions on AWS.
• Develop ingestion pipelines from multiple sources (batch and streaming).
• Implement data transformation, enrichment, cleansing, and validation processes.

Skills

PySpark

Python

SQL

Data Modeling

Data Quality

Batch & RealTime Processing

Data Warehousing Concepts

ETL/ELT Design

Education

Bachelor's or Master's in Computer Science / Engineering / Information Systems

AWS Certified Data Analytics or AWS Solution Architect (Preferred)

Tools

Databricks

Databricks Lakehouse Architecture

Databricks Workflows

Unity Catalog

Databricks SQL

Auto Loader

Structured Streaming

Terraform

AWS

Amazon S3

AWS Glue

AWS Lambda

AWS IAM

Amazon Redshift

AWS CloudWatch

EventBridge

KMS

Secrets Manager

Job description

• We are seeking a highly skilled Databricks Engineer with AWS expertise to support Data Services portfolio
• The role will be responsible for designing developing optimizing and supporting enterprise scale data platforms data pipelines and analytics solutions leveraging Databricks Lakehouse AWS Cloud and modern data engineering practices
• The candidate will work closely with data architects' application teams business stakeholders and platform engineering teams to deliver scalable secure and high-performance data solutions

Key Responsibilities

• Design develop and maintain scalable data pipelines using Databricks Pyspark Spark SQL and Delta Lake
• Build and support enterprise data lake and Lakehouse solutions on AWS
• Develop batch and Realtime ingestion frameworks from multiple source systems
• Implement data transformation enrichment cleansing and validation processes
• Optimize Spark workloads for performance scalability and cost efficiency
• Design and implement data models supporting reporting analytics and downstream applications
• Integrate Databricks with AWS services including S3 Glue Lambda Redshift EventBridge and IAM
• Support data migration and modernization initiatives from traditional data warehouses to Databricks Lakehouse
• Implement CICD pipelines and Infrastructure as Code IaC using Terraform and Gitbased repositories
• Collaborate with security teams to ensure compliance with client governance risk and data security standards
• Troubleshoot production incidents and provide L3 operational support
• Participate in architecture reviews code reviews and technical governance activities

Mandatory Technical Skills

• Databricks
• Databricks Lakehouse Architecture
• Databricks Workflows
• Unity Catalog
• Databricks SQL
• Auto Loader
• Structured Streaming
• Databricks Performance Tuning
• Medallion Architecture
• AWS
• Amazon S3
• AWS Glue
• AWS Lambda
• AWS IAM
• Amazon Redshift
• AWS CloudWatch
• SNS SQS
• Event Bridge
• KMS
• Secrets Manager
• Pyspark
• Python
• SQL
• Data Warehousing Concepts
• ETLELT Design
• Data Modeling
• Data Quality Frameworks
• Batch RealTime Processing

Preferred Skills

• Snowflake
• Microsoft Fabric
• Kafka
• Airflow
• dbt
• Terraform
• Docker
• Informatica
• Collibra
• Data Vault Modeling
• Dimensional Data Modeling
• Banking Domain Experience
• Strong experience within Banking and Financial Services
• Understanding of enterprise data governance regulatory reporting risk compliance and audit requirements
• Experience working in large global organizations with stringent security and operational standards

Qualifications

• Bachelors or Masters degree in Computer Science Engineering Information Systems or related discipline
• AWS Certified Data Analytics or AWS Solution Architect Preferred