Databricks Data Engineering Lead | AWS Lakehouse Architect

Ethereum Technologies; City of Albany (NY); On-site; USD 99,000 - 141,000; Full time

Job summary

Ethereum Technologies in Albany, NY seeks an experienced Databricks Data Engineering Lead to design and build scalable data solutions using Databricks, Python, PySpark, Delta Lake, and Databricks Genie. 8+ years of data engineering experience required.

You will lead the engineering team, set architecture direction, mentor engineers, and collaborate with stakeholders to deliver high-performance, secure data platforms on AWS.

Qualifications

• 8+ years in Data Engineering / Data Platforms.
• Hands-on Databricks experience with Genie.
• Python and PySpark expert.
• Delta Lake and Lakehouse architecture knowledge.
• SQL and data modelling proficiency.
• Experience building batch/streaming pipelines.

Responsibilities

• Lead design and development of enterprise-scale data solutions on Databricks.
• Design scalable ETL/ELT pipelines with Python and PySpark.
• Provide hands-on technical leadership for Genie implementations.
• Architect Delta Lake and Lakehouse-based data platforms.
• Build reusable data ingestion, transformation, validation components.
• Develop optimized PySpark workloads for performance and reliability.
• Enforce data quality, governance, security, monitoring, and ops controls.
• Lead architecture/code reviews and engineering best practices.
• Mentor data engineers and guide implementation teams.
• Collaborate with architects, analysts, and application teams to translate requirements.
• Support production deployments, tuning, incidents, and continuous improvement.

Skills

Databricks

Databricks Genie

Python

PySpark

Delta Lake

Lakehouse

SQL

Data modelling

ETL/ELT pipelines

Data quality

Observability

Production support

Technical leadership

Tools

AWS

S3

Glue

Redshift

CloudWatch

Job description

Ethereum Technologies in Albany, NY seeks an experienced Databricks Data Engineering Lead to design and build scalable data solutions using Databricks, Python, PySpark, Delta Lake, and Databricks Genie. 8+ years of data engineering experience required.

You will lead the engineering team, set architecture direction, mentor engineers, and collaborate with stakeholders to deliver high-performance, secure data platforms on AWS.