Senior Databricks Data Engineer | AWS Pipelines, FISMA High

shinvestmentsllc

United States

Remote

USD 120,000 - 170,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Medical, dental, vision
401(k) plan

Job summary

shinvestmentsllc in the United States seeks a Data Engineer/Databricks Developer to design, build, and maintain scalable data pipelines on AWS Databricks, aligned with FISMA High and multi-tenant governance. You will work across ingestion, transformation, and delivery of analytics-ready datasets, implementing Bronze-Silver-Gold pipelines, and collaborating with Agile teams.

Proficiency in PySpark, Python, SQL, and Delta Lake is required, with experience in DMS and CI/CD using GitLab to support

Qualifications

  • Proven hands-on experience building production-grade data pipelines as a Data Engineer or Databricks Developer.
  • Strong working knowledge of Databricks on AWS, including E2 architecture, cluster configuration, job orchestration, and workspace management.
  • Hands-on experience with Apache Spark, PySpark, and Pandas for distributed data processing.
  • Solid programming skills in Python and advanced SQL including complex joins, window functions, and aggregations.
  • Practical experience with AWS DMS for change data capture and replication, plus Delta Lake techniques such as OPTIMIZE, Z-ORDER, and VACUUM.
  • Experience with GitLab CI/CD pipelines and working in Agile or Scrum environments.

Responsibilities

  • Analyze and collect data from relational databases, APIs, external providers, and real-time streaming sources.
  • Design and implement scalable pipelines that cleanse, transform, and aggregate data for downstream reporting and analytics.
  • Build and maintain Databricks solutions using the medallion architecture across Bronze, Silver, and Gold layers.
  • Produce source-to-target mapping documentation and conduct unit testing to validate pipeline logic and accuracy.
  • Implement real-time and near-real-time ingestion using AWS DMS and other AWS tools.
  • Develop transformation logic with PySpark and Pandas, manage versioning and CI/CD through GitLab, and align practices with FISMA High and multi-tenant governance requirements.

Skills

Python programming
Advanced SQL
Agile/Scrum experience
Data pipeline design

Tools

Databricks on AWS
Apache Spark
PySpark
Pandas
Delta Lake
GitLab CI/CD
AWS DMS
CloudFormation/Terraform

Job description

shinvestmentsllc in the United States seeks a Data Engineer/Databricks Developer to design, build, and maintain scalable data pipelines on AWS Databricks, aligned with FISMA High and multi-tenant governance. You will work across ingestion, transformation, and delivery of analytics-ready datasets, implementing Bronze-Silver-Gold pipelines, and collaborating with Agile teams.

Proficiency in PySpark, Python, SQL, and Delta Lake is required, with experience in DMS and CI/CD using GitLab to support

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

shinvestmentsllc • United States

Remote
USD 120,000 - 170,000
Medical, dental, vision
401(k) plan
Senior Databricks Engineer – Remote Data Pipelines
Senior Databricks Engineer – Remote Data Pipelines

Princeton IT Services, Inc • United States

Remote
USD 130,000 - 180,000
Senior Databricks Engineer - AWS Spark & Data Pipelines
Senior Databricks Engineer - AWS Spark & Data Pipelines

Red Cedar Consultancy • Washington

On-site
USD 140,000 - 180,000
Senior Databricks Data Engineer — Build Scalable Pipelines
Senior Databricks Data Engineer — Build Scalable Pipelines

Remote • San Francisco (CA)

On-site
USD 140,000 - 190,000
Health and dental insurance
Meal and food allowance
Childcare assistance
+2
Senior Databricks Data Engineer: AI DataOps & Pipelines
Senior Databricks Data Engineer: AI DataOps & Pipelines

Ciandt • United States

Remote
BRL 180,000 - 330,000
Health insurance
Meal allowance
Life insurance
+6
Senior Databricks Data Engineer for AI-Powered Pipelines
Senior Databricks Data Engineer for AI-Powered Pipelines

CI&T • United States

Remote
USD 120,000 - 160,000
Premium Healthcare
Meal voucher
Maternity and Parental leaves
+5
Senior Databricks Data Engineer | PySpark & Cloud Pipelines
Senior Databricks Data Engineer | PySpark & Cloud Pipelines

Valiant Technologies LLC • Madison (WI)

On-site
USD 120,000 - 150,000
Senior Databricks Data Engineer - Spark, PySpark & AWS
Senior Databricks Data Engineer - Spark, PySpark & AWS

Ethereum Technologies • Dallas (TX)

On-site
USD 120,000 - 170,000
Senior Data Engineer - Databricks/PySpark/SQL | Remote
Senior Data Engineer - Databricks/PySpark/SQL | Remote

HumanIT Digital Consulting • United States

Remote
USD 28,000 - 37,000
15th month salary
Health insurance for family
Birthday off
+3
Databricks Data Engineer Lead: Spark & ETL on AWS
Databricks Data Engineer Lead: Spark & ETL on AWS

Red Cedar Consultancy, LLC • Washington

On-site
USD 110,000 - 115,000
401(k) matching
Dental insurance
Health insurance
+5