Data Engineer

MACHINE LEARNING TECHNOLOGIES LLC

Wilmington (DE)

On-site

USD 55,000 - 96,000

Full time

31 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

MACHINE LEARNING TECHNOLOGIES LLC is seeking a hands-on Data Engineer in Wilmington, DE to build and run end-to-end data pipelines on AWS using DMS and Databricks. You will own pipelines from source ingestion through transformation to analytics-ready data, with emphasis on CDC, upserts, and schema changes.

You'll work directly with the customer, deliver on AWS with no ramp-up time, and contribute to performance tuning, data quality, and runbook documentation.

Qualifications

  • 6+ years in data engineering, with at least 3 years on AWS data services and Databricks.
  • Hands-on experience building end-to-end data pipelines from source to curated data.
  • Proficient in PySpark, Python, and SQL for production-grade pipelines.

Responsibilities

  • Design, build, and maintain end-to-end data pipelines from source ingestion through transformation to curated outputs.
  • Configure and run AWS DMS replication tasks for full loads and CDC from Oracle, SQL Server, and MariaDB.
  • Land DMS output in Amazon S3 and process in Databricks using Delta Lake and medallion architecture.
  • Write complex SQL for transformations, validation, reconciliation, and troubleshooting.
  • Handle CDC logic: upserts, merges, deletes, late-arriving data, and schema changes.
  • Tune Spark jobs and DMS tasks for performance, cost, and reliability; diagnose replication lag and failures.
  • Build data quality checks, logging, alerting, and restart/recovery so pipelines run autonomously.
  • Orchestrate workloads with Databricks Workflows and keep code under source control with peer review.
  • Meet with customer stakeholders to gather requirements, explain design choices, and present issue resolutions.
  • Write clear technical documentation for pipelines, data flows, and runbooks.

Skills

AWS DMS
Databricks
PySpark
Python
SQL
Data pipelines
Communication
AWS

Tools

Delta Lake
S3
Databricks Jobs
Git
Terraform

Job description

Job Overview
  • Job ID: J55442
  • Job Title: Data Engineer
  • Location: Wilmington, DE
  • Duration: 14 Months + Extension
  • Domain Exposure: Aerospace/ Airlines
  • Client: To Be Discussed Later
  • Employment Type: W-2, 1099, C2C
  • Work Authorization: US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other
  • Bill Rate: $79/hour (C2C)

A hands-on Senior Data Engineer to build and run end-to-end data pipelines on AWS DMS and Databricks using PySpark, Python, and SQL. The position owns pipelines from source-database replication through transformation to curated, analytics-ready data, working directly with the customer. Immediate delivery on AWS is required with no ramp-up time from Azure or another cloud.

Key Responsibilities
  • Design, build, and maintain end-to-end data pipelines, from source ingestion through transformation to curated output tables.
  • Configure and run AWS DMS replication tasks for full loads and ongoing Change Data Capture (CDC) from relational sources such as Oracle, SQL Server, and MariaDB.
  • Land DMS output in Amazon S3 and process it in Databricks with PySpark, using Delta Lake and a bronze/silver/gold (medallion) design.
  • Write complex, efficient SQL for transformations, data validation, reconciliation, and troubleshooting.
  • Handle CDC logic: upserts and merges, deletes, late-arriving data, schema changes, and deduplication.
  • Tune Spark jobs and DMS tasks for performance, cost, and reliability; diagnose replication lag and job failures.
  • Build in data quality checks, logging, alerting, and restart/recovery so pipelines run without hand-holding.
  • Orchestrate workloads with Databricks Workflows (or similar) and keep code in source control with peer review.
  • Meet regularly with customer stakeholders: gather requirements, explain design choices and trade-offs, give status updates, and walk through issues and fixes.
  • Write clear technical documentation for pipelines, data flows, and runbooks.
Required Qualifications (Must Have)
  • All experience below must be hands-on and on AWS (experience only on Azure without AWS does not meet the bar):
  • AWS DMS: Personally set up, run, and troubleshot DMS replication (full load + CDC) in a production setting, including endpoints, replication instances, task settings, and table mappings.
  • Databricks: Strong hands-on development on Databricks on AWS: notebooks and jobs, Delta Lake, MERGE/upsert patterns, cluster configuration, and performance tuning.
  • PySpark and Python: Writes production-grade PySpark transformations and Python code (modular, tested, version-controlled).
  • End-to-end pipelines: Has built complex pipelines from source to curated data, not only the reporting or visualization layer.
  • SQL: Strong SQL, including joins across large tables, window functions, CTEs, query tuning, and data reconciliation.
  • Communication: Can explain work clearly to technical and non-technical audiences, and comfortable leading customer conversations.
  • Experience: Typically 6+ years in data engineering, with at least 3 years currently on AWS data services and Databricks.
Nice To Have
  • AWS Glue: Jobs, crawlers, Data Catalog
  • Power BI: Enough to understand how downstream reports consume the data
  • Other AWS services: S3, IAM, Lambda, Step Functions, CloudWatch, Redshift
  • Databricks features: Unity Catalog, Delta Live Tables, or Auto Loader
  • CI/CD for data code: Git, Databricks Asset Bundles, Terraform
  • Certifications: Databricks or AWS certifications (e.g., Databricks Data Engineer Professional, AWS Data Engineer Associate)
  • Sector Experience: Experience on federal or public‑sector projects
Who This Role Is Not For
  • DevOps or platform engineers whose DMS/Databricks work is mostly infrastructure, provisioning, or CI/CD, not building pipelines.
  • Reporting or BI developers whose main work is dashboards, Power BI, or report‑layer SQL.
  • Azure‑first engineers who would need to learn AWS on the job.
  • Glue‑only engineers whose AWS pipeline experience is mainly AWS Glue, without strong DMS and Databricks.
Equal Opportunity Employer

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

Robotics Technologies • Wilmington (DE)

On-site
USD 120,000 - 180,000
Data Engineer
Data Engineer

Quantum Technologies LLC • Wilmington (DE)

On-site
USD 140,000 - 210,000
Data Engineer
Data Engineer

Valiant Technologies LLC • Wilmington (DE)

On-site
USD 140,000 - 200,000
Data Engineer
Data Engineer

Robotic Process Automation LLC • Wilmington (DE)

On-site
USD 140,000 - 200,000
Data Engineer
Data Engineer

CloudBigD • Wilmington (DE)

On-site
USD 120,000 - 180,000
Data Engineer
Data Engineer

Ethereum Technologies LLC • Wilmington (DE)

On-site
USD 140,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 150,000 - 190,000
Senior Databricks Developer
Senior Databricks Developer

Quantum Technologies LLC • Pennsylvania

On-site
USD 96,000 - 131,000
Senior Databricks Developer
Senior Databricks Developer

Valiant Technologies LLC • Pennsylvania

On-site
USD 92,000 - 131,000
Senior Databricks Developer
Senior Databricks Developer

Robotic Process Automation LLC • Pennsylvania

On-site
USD 95,000 - 128,000