Senior Data Engineer: End-to-End AWS DMS + Databricks

MACHINE LEARNING TECHNOLOGIES LLC

Wilmington (DE)

On-site

USD 55,000 - 96,000

Full time

31 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

MACHINE LEARNING TECHNOLOGIES LLC is seeking a hands-on Data Engineer in Wilmington, DE to build and run end-to-end data pipelines on AWS using DMS and Databricks. You will own pipelines from source ingestion through transformation to analytics-ready data, with emphasis on CDC, upserts, and schema changes.

You'll work directly with the customer, deliver on AWS with no ramp-up time, and contribute to performance tuning, data quality, and runbook documentation.

Qualifications

  • 6+ years in data engineering, with at least 3 years on AWS data services and Databricks.
  • Hands-on experience building end-to-end data pipelines from source to curated data.
  • Proficient in PySpark, Python, and SQL for production-grade pipelines.

Responsibilities

  • Design, build, and maintain end-to-end data pipelines from source ingestion through transformation to curated outputs.
  • Configure and run AWS DMS replication tasks for full loads and CDC from Oracle, SQL Server, and MariaDB.
  • Land DMS output in Amazon S3 and process in Databricks using Delta Lake and medallion architecture.
  • Write complex SQL for transformations, validation, reconciliation, and troubleshooting.
  • Handle CDC logic: upserts, merges, deletes, late-arriving data, and schema changes.
  • Tune Spark jobs and DMS tasks for performance, cost, and reliability; diagnose replication lag and failures.
  • Build data quality checks, logging, alerting, and restart/recovery so pipelines run autonomously.
  • Orchestrate workloads with Databricks Workflows and keep code under source control with peer review.
  • Meet with customer stakeholders to gather requirements, explain design choices, and present issue resolutions.
  • Write clear technical documentation for pipelines, data flows, and runbooks.

Skills

AWS DMS
Databricks
PySpark
Python
SQL
Data pipelines
Communication
AWS

Tools

Delta Lake
S3
Databricks Jobs
Git
Terraform

Job description

MACHINE LEARNING TECHNOLOGIES LLC is seeking a hands-on Data Engineer in Wilmington, DE to build and run end-to-end data pipelines on AWS using DMS and Databricks. You will own pipelines from source ingestion through transformation to analytics-ready data, with emphasis on CDC, upserts, and schema changes.

You'll work directly with the customer, deliver on AWS with no ramp-up time, and contribute to performance tuning, data quality, and runbook documentation.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer: AWS DMS + Databricks
Senior Data Engineer: AWS DMS + Databricks

Robotics Technologies • Wilmington (DE)

On-site
USD 120,000 - 180,000
Senior AWS Data Engineer - Databricks & DMS Pipelines
Senior AWS Data Engineer - Databricks & DMS Pipelines

CloudBigD • Wilmington (DE)

On-site
USD 120,000 - 180,000
Senior AWS Data Engineer – DMS & Databricks Expert
Senior AWS Data Engineer – DMS & Databricks Expert

Ethereum Technologies LLC • Wilmington (DE)

On-site
USD 140,000 - 180,000
Senior Data Engineer - AWS DMS & Databricks Lead
Senior Data Engineer - AWS DMS & Databricks Lead

Quantum Technologies LLC • Wilmington (DE)

On-site
USD 140,000 - 210,000
Data Engineer
Data Engineer

MACHINE LEARNING TECHNOLOGIES LLC • Wilmington (DE)

On-site
USD 55,000 - 96,000
Senior AWS Data Engineer - DMS + Databricks Pipelines
Senior AWS Data Engineer - DMS + Databricks Pipelines

Valiant Technologies LLC • Wilmington (DE)

On-site
USD 140,000 - 200,000
Senior Data Engineer - AWS DMS & Databricks
Senior Data Engineer - AWS DMS & Databricks

Robotic Process Automation LLC • Wilmington (DE)

On-site
USD 140,000 - 200,000
Data Engineer
Data Engineer

Robotics Technologies • Wilmington (DE)

On-site
USD 120,000 - 180,000
Data Engineer
Data Engineer

CloudBigD • Wilmington (DE)

On-site
USD 120,000 - 180,000
Data Engineer
Data Engineer

Robotic Process Automation LLC • Wilmington (DE)

On-site
USD 140,000 - 200,000