Senior Data Engineer, Databricks & Lakehouse Expert

Techgene Solutions LLC; Pasadena (CA); Hybrid; USD 120,000 - 170,000; Full time

Job summary

Techgene Solutions LLC in Pasadena, CA seeks a Sr. Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python. You will build ETL/ELT workflows, integrate data from multiple sources, and ensure data quality and governance.

The role requires hands-on expertise with Delta Lake, Spark performance tuning, and cloud platforms (AWS/Azure/GCP). You will collaborate with data architects and scientists, implement CI/CD, and support production data

Qualifications

• Hands-on Databricks, PySpark, and Python skills.
• Experience building ETL/ELT pipelines across data sources.
• Strong SQL skills and Delta Lake knowledge.
• Experience with data lake/lakehouse architectures and Spark tuning.
• CI/CD and Git experience for data pipelines.
• Cloud exposure on AWS, Azure, or GCP.

Responsibilities

• Design, develop, and maintain scalable data pipelines with Databricks, PySpark, and Python.
• Build ETL/ELT workflows to ingest, transform, cleanse, and integrate data from multiple sources.
• Optimize Spark jobs and performance; work with Delta Lake for storage and versioning.
• Collaborate with data architects, scientists, BI developers and stakeholders.
• Implement CI/CD and source-control practices for data engineering code.
• Troubleshoot production data pipelines and perform RCA when failures occur.
• Ensure data security, governance, lineage, and compliance.

Skills

Databricks

PySpark

Python

ETL/ELT pipelines

SQL

Delta Lake

Data lake/lakehouse

Spark performance tuning

Git

CI/CD

AWS

Azure

GCP

Tools

Airflow

Terraform

Unity Catalog

Kafka

Job description

Techgene Solutions LLC in Pasadena, CA seeks a Sr. Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python. You will build ETL/ELT workflows, integrate data from multiple sources, and ensure data quality and governance.

The role requires hands-on expertise with Delta Lake, Spark performance tuning, and cloud platforms (AWS/Azure/GCP). You will collaborate with data architects and scientists, implement CI/CD, and support production data