Databricks Data Engineer

Ethereum Technologies LLC; Harrisburg (Dauphin County); On-site; USD 85,000 - 121,000; Part time

Job summary

Ethereum Technologies in Harrisburg, PA seeks an experienced Databricks Data Engineer to design and implement a scalable Sales Data Platform on Databricks (AWS). The role is hands-on and architecture-driven, focusing on ingestion, transformation, modelling, and optimization using modern Lakehouse patterns.

You will define end-to-end Lakehouse architecture, establish data modelling standards for Sales domains, and drive performance, scalability, and cost optimisation for large-scale pipelines.

Qualifications

• 8+ years in Data Engineering / Data Architecture roles.
• 4+ years hands-on experience with Databricks.
• Strong expertise in PySpark & Spark SQL.
• Deep experience with Delta Lake.
• Strong knowledge of AWS cloud services (S3, IAM, Glue, CloudWatch).
• Strong understanding of: Fact & dimension modelling; Slowly Changing Dimensions (SCD Type 1 / 2); Large-scale data processing patterns.
• Experience handling high-volume, high-velocity datasets.

Responsibilities

• Lead design and implementation of a scalable Sales Data Platform on Databricks (AWS).
• Hands-on role focused on data ingestion, transformation, modelling, and optimization using Lakehouse patterns.
• Define end-to-end Lakehouse architecture on Databricks (AWS) for Sales data domains.
• Establish data modelling standards for Sales facts, dimensions, hierarchies, and aggregations.
• Drive performance, scalability, and cost optimisation for pipelines.
• Build PySpark-based data pipelines in Databricks and ensure fault-tolerance.

Skills

8+ years data engineering

4+ years Databricks hands-on

PySpark & Spark SQL

Delta Lake

AWS services (S3, IAM, Glue, Cloud1, 0

Tools

Databricks

PySpark

Spark SQL

Delta Lake

AWS Glue

Amazon S3

CloudWatch

IAM

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title :Databricks Data Engineer
Job Type : C2C/1099 Contract
Location : Harrisburg, PA
Duration :10 months
Bill Rate : $75/Hr on c2c

Role Summary

• We are seeking a Databricks Engineer to lead the design and implementation of a scalable Sales Data Platform as part of the One Data initiative on Databricks running on AWS.
• The role is hands‑on and architecture‑driven, focused on data ingestion, transformation, modelling, and optimization using modern Lakehouse patterns.
• The architect will work closely with data engineers, source system teams, and downstream consumers to deliver high‑quality, governed, and performance‑optimised sales datasets.
• Define end‑to‑end Lakehouse architecture on Databricks (AWS) for Sales data domains.
• Design medallion architecture (Bronze / Silver / Gold) aligned with One Data standards.
• Establish data modelling standards for Sales facts, dimensions, hierarchies, and aggregations.
• Define scalable ingestion patterns for batch and incremental loads.
• Drive performance, scalability, and cost optimisation best practices.
• Build and guide development of PySpark‑based data pipelines in Databricks.
• ACID transactions.
• Design and optimise large‑scale joins, aggregations, and window functions.
• Implement CDC and incremental processing using watermarking and change detection.
• Ensure idempotent, restart‑able, and fault‑tolerant pipelines.

AWS & Platform Integration

• Architect solutions using AWS services:
  • Amazon S3 (data lake storage).
  • IAM (security & access control).
  • AWS Glue / Glue CatLog.
  • CloudWatch (monitoring & logging).
• Optimize Databricks cluster configurations (job vs all‑purpose clusters).
• Implement secrets management and secure connectivity patterns.

Data Quality, Governance & Reliability

• Define and implement data quality checks and validations.
• Ensure data lineage and metadata capture.
• Implement error handling, auditing, and reconciliation frameworks.
• Support data governance and access control requirements.

DevOps & Operational Excellence

• Implement CI/CD pipelines for Databricks notebooks and jobs.
• Enforce code versioning, reviews, and deployment standards.
• Design monitoring, alerting, and SLA tracking for pipelines.
• Support production stabilisation and performance tuning.
• Act as technical lead / mentor for Databricks data engineers.
• Collaborate with:
  • Source system teams (Sales, CRM, ERP).
  • Data consumers (analytics, downstream apps).
  • Cloud/platform teams.
• Translate business requirements into robust technical designs.

Required Skills & Experience

Core Technical Skills

• 8+ years in Data Engineering / Data Architecture roles.
• 4+ years hands‑on experience with Databricks.
• Strong expertise in PySpark & Spark SQL.
• Deep experience with Delta Lake.
• Strong knowledge of AWS cloud services (S3, IAM, Glue, CloudWatch).
• Strong understanding of:
  • Fact & dimension modelling.
  • Slowly Changing Dimensions (SCD Type 1 / 2).
  • Large‑scale data processing patterns.
• Experience handling high‑volume, high‑velocity datasets.

Platform & Operational Skills

• Databricks job orchestration and scheduling.
• Cluster sizing and performance tuning.
• CI/CD for data platforms.
• Strong troubleshooting and debugging skills.

Nice‑to‑Have

• Experience with enterprise One Data / Data Mesh programs.
• Exposure to real‑time or near‑real‑time ingestion patterns.
• AWS certifications or Databricks certifications.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.