Sr. Data Engineer

Techgene Solutions LLC

Pasadena (CA)

Hybrid

USD 120,000 - 170,000

Full time

11 days ago

Application generator

An application made for this job — a tailored resume and cover letter that speak straight to the posting.

Get past ATS filters

Job summary

Techgene Solutions LLC in Pasadena, CA seeks a Sr. Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python. You will build ETL/ELT workflows, integrate data from multiple sources, and ensure data quality and governance.

The role requires hands-on expertise with Delta Lake, Spark performance tuning, and cloud platforms (AWS/Azure/GCP). You will collaborate with data architects and scientists, implement CI/CD, and support production data

Qualifications

Hands-on Databricks, PySpark, and Python skills.
Experience building ETL/ELT pipelines across data sources.
Strong SQL skills and Delta Lake knowledge.
Experience with data lake/lakehouse architectures and Spark tuning.
CI/CD and Git experience for data pipelines.
Cloud exposure on AWS, Azure, or GCP.

Responsibilities

Design, develop, and maintain scalable data pipelines with Databricks, PySpark, and Python.
Build ETL/ELT workflows to ingest, transform, cleanse, and integrate data from multiple sources.
Optimize Spark jobs and performance; work with Delta Lake for storage and versioning.
Collaborate with data architects, scientists, BI developers and stakeholders.
Implement CI/CD and source-control practices for data engineering code.
Troubleshoot production data pipelines and perform RCA when failures occur.
Ensure data security, governance, lineage, and compliance.

Skills

Databricks

PySpark

Python

ETL/ELT pipelines

SQL

Delta Lake

Data lake/lakehouse

Spark performance tuning

Git

CI/CD

AWS

Azure

GCP

Tools

Airflow

Terraform

Unity Catalog

Kafka

Job description

Role : Sr. Data Engineer

Location: Pasadena, CA

Work Arrangement: Hybrid

Job Summary

We are looking for an experienced Data Engineer with strong expertise in Databricks, PySpark, and Python to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands‑on experience building ETL/ELT pipelines, data processing frameworks, and data lake/lakehouse solutions using Databricks and cloud technologies.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
Develop ETL/ELT workflows to ingest, transform, cleanse, and integrate data from multiple sources.
Build and optimize data processing jobs using PySpark and Spark SQL.
Work extensively with Databricks Lakehouse, Delta Lake, notebooks, workflows, and clusters.
Develop reusable Python modules and frameworks for data processing and automation.
Implement data quality checks, validation, error handling, and monitoring within data pipelines.
Optimize Spark jobs, including partitioning, caching, joins, and performance tuning.
Work with Delta Lake for data storage, transformation, versioning, and incremental processing.
Integrate data from relational databases, APIs, files, cloud storage, and other enterprise data sources.
Collaborate with Data Architects, Data Scientists, BI Developers, and business stakeholders to understand data requirements.
Implement CI/CD and source-control practices for data engineering code.
Troubleshoot production data pipeline failures and perform root cause analysis (RCA).
Ensure data security, governance, lineage, and compliance requirements are followed.
Participate in design discussions, code reviews, testing, deployment, and production support.

Required Skills