Senior Databricks Data Engineer - AWS Lakehouse

Seneca Holdings; Buffalo (NY); On-site; USD 110,000 - 140,000; Full time

Job summary

Seneca Holdings is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines within a Databricks E2 environment on AWS. The role focuses on end-to-end data lifecycle—from ingestion to analytics-ready datasets—under FISMA High and multi-tenant security.

The ideal candidate will have hands-on Databricks, Spark, and AWS experience, plus Python, SQL, and CI/CD with GitLab. Collaboration in an Agile setting is essential for delivering reliable data platforms.

Qualifications

• Hands-on experience building production-grade data pipelines.
• Strong knowledge of Databricks on AWS (E2, cluster config, orchestration).
• Experience with Apache Spark, PySpark and Pandas at scale.
• Proficient in Python; R is a plus.
• Experience with Delta Lake and data lakehouse concepts.
• Familiar with GitLab CI/CD pipelines.
• Agile/Scrum project experience.

Responsibilities

• Design, build, and maintain scalable data pipelines in Databricks.
• Implement medallion architecture (Bronze/Silver/Gold) for datasets.
• Write advanced SQL for aggregations and data quality checks.
• Ingest data with AWS DMS and other AWS services in real time.
• Collaborate in Agile environment and ensure security/compliance.
• Develop source-to-target mapping docs and perform unit tests.

Skills

Databricks on AWS

Python

PySpark

Pandas

SQL

GitLab CI/CD

Agile/Scrum

Tools

Delta Lake

DMS

RDS

Job description

Seneca Holdings is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines within a Databricks E2 environment on AWS. The role focuses on end-to-end data lifecycle—from ingestion to analytics-ready datasets—under FISMA High and multi-tenant security.

The ideal candidate will have hands-on Databricks, Spark, and AWS experience, plus Python, SQL, and CI/CD with GitLab. Collaboration in an Agile setting is essential for delivering reliable data platforms.