Senior Data Engineer

shinvestmentsllc; United States; Remote; USD 120,000 - 170,000; Full time

Benefits offered by this job

Medical, dental, vision

401(k) plan

Job summary

shinvestmentsllc in the United States seeks a Data Engineer/Databricks Developer to design, build, and maintain scalable data pipelines on AWS Databricks, aligned with FISMA High and multi-tenant governance. You will work across ingestion, transformation, and delivery of analytics-ready datasets, implementing Bronze-Silver-Gold pipelines, and collaborating with Agile teams.

Proficiency in PySpark, Python, SQL, and Delta Lake is required, with experience in DMS and CI/CD using GitLab to support

Qualifications

• Proven hands-on experience building production-grade data pipelines as a Data Engineer or Databricks Developer.
• Strong working knowledge of Databricks on AWS, including E2 architecture, cluster configuration, job orchestration, and workspace management.
• Hands-on experience with Apache Spark, PySpark, and Pandas for distributed data processing.
• Solid programming skills in Python and advanced SQL including complex joins, window functions, and aggregations.
• Practical experience with AWS DMS for change data capture and replication, plus Delta Lake techniques such as OPTIMIZE, Z-ORDER, and VACUUM.
• Experience with GitLab CI/CD pipelines and working in Agile or Scrum environments.

Responsibilities

• Analyze and collect data from relational databases, APIs, external providers, and real-time streaming sources.
• Design and implement scalable pipelines that cleanse, transform, and aggregate data for downstream reporting and analytics.
• Build and maintain Databricks solutions using the medallion architecture across Bronze, Silver, and Gold layers.
• Produce source-to-target mapping documentation and conduct unit testing to validate pipeline logic and accuracy.
• Implement real-time and near-real-time ingestion using AWS DMS and other AWS tools.
• Develop transformation logic with PySpark and Pandas, manage versioning and CI/CD through GitLab, and align practices with FISMA High and multi-tenant governance requirements.

Skills

Python programming

Advanced SQL

Agile/Scrum experience

Data pipeline design

Tools

Databricks on AWS

Apache Spark

PySpark

Pandas

Delta Lake

GitLab CI/CD

AWS DMS

CloudFormation/Terraform

Job description

Role overview

This position supports a federal government contracting team by designing, building, and maintaining scalable data pipelines within a Databricks environment hosted on AWS. The work operates under FISMA High and multi-tenant security and compliance standards, supporting ingestion, transformation, and delivery of analytics-ready datasets across the full data lifecycle.

Responsibilities

• Analyze and collect data from relational databases, APIs, external providers, and real-time streaming sources
• Design and implement scalable pipelines that cleanse, transform, and aggregate data for downstream reporting and analytics
• Build and maintain Databricks solutions using the medallion architecture across Bronze, Silver, and Gold layers
• Produce source-to-target mapping documentation and conduct unit testing to validate pipeline logic and accuracy
• Implement real-time and near-real-time ingestion using AWS Database Migration Service and other AWS tools
• Develop transformation logic with PySpark and Pandas, manage versioning and CI/CD through GitLab, and align practices with FISMA High and multi-tenant governance requirements

Requirements

• Proven hands‑on experience building production‑grade data pipelines as a Data Engineer or Databricks Developer
• Strong working knowledge of Databricks on AWS, including E2 architecture, cluster configuration, job orchestration, and workspace management
• Hands‑on experience with Apache Spark, PySpark, and Pandas for distributed data processing
• Solid programming skills in Python and advanced SQL including complex joins, window functions, and aggregations
• Practical experience with AWS DMS for change data capture and replication, plus Delta Lake techniques such as OPTIMIZE, Z-ORDER, and VACUUM
• Experience with GitLab CI/CD pipelines and working in Agile or Scrum environments

Nice to have

• Familiarity with additional AWS services such as S3, Glue, Lambda, Step Functions, CloudWatch
• Experience operating within FISMA High, FedRAMP, or other regulated multi‑tenant environments
• Exposure to infrastructure‑as‑code tooling like Terraform or CloudFormation, plus streaming technologies such as Kafka, Kinesis, or Spark Structured Streaming

Benefits and work setup

• Comprehensive benefits including medical, dental, vision, life, and disability coverage, voluntary programs, health savings and flexible spending accounts, and a 401(k) plan
• Eligibility for full‑time employees working at least 30 hours per week