Senior Data Engineer

Topcoder; California; On-site; USD 120,000 - 170,000; Full time

Job summary

Topcoder is seeking a data engineering specialist to build and optimize Databricks pipelines that transform raw data into curated, business-ready layers. You will own production deployments through Azure DevOps, including code reviews, testing, and releases, and track work in ADO Boards.

You will monitor and tune jobs for performance, reliability, and cost, implement data quality checks, and collaborate with business users and the Platform team to translate requirements into robust technical

Qualifications

• Strong hands-on experience with Databricks (notebooks, jobs, workflows, clusters).
• Proficiency in PySpark, Python, and advanced SQL.
• Solid understanding of Delta Lake, data modeling, and ETL/ELT patterns.
• Experience supporting production workloads, with strong debugging and data validation skills.
• Working knowledge of Git and CI/CD, ideally Azure DevOps (Repos, Pipelines, Boards).

Responsibilities

• Build and optimize Databricks pipelines (PySpark, SQL, Delta Lake) that transform Raw data into curated and business-ready layers.
• Own production deployments through Azure DevOps (ADO), including code reviews, testing, and releases; track work in ADO Boards.
• Monitor, troubleshoot, and tune jobs for performance, reliability, and cost.
• Implement data quality checks and governance using Unity Catalog.
• Partner with business users and the Platform team to turn requirements into technical solutions.
• Maintain clear documentation and support runbooks.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

Data modeling

ETL/ELT

Git

Azure DevOps

Tools

Azure DevOps

Git

Job description

Build and optimize Databricks pipelines (PySpark, SQL, Delta Lake) that transform Raw data into curated and business-ready layers. Own production deployments through Azure DevOps (ADO), including code reviews, testing, and releases; track work in ADO Boards. Monitor, troubleshoot, and tune jobs for performance, reliability, and cost. Implement data quality checks and governance using Unity Catalog. Partner with business users and the Platform team to turn requirements into technical solutions. Maintain clear documentation and support runbooks.

Responsibilities

• Build and optimize Databricks pipelines (PySpark, SQL, Delta Lake) that transform Raw data into curated and business-ready layers.
• Own production deployments through Azure DevOps (ADO), including code reviews, testing, and releases; track work in ADO Boards.
• Monitor, troubleshoot, and tune jobs for performance, reliability, and cost.
• Implement data quality checks and governance using Unity Catalog.
• Partner with business users and the Platform team to turn requirements into technical solutions.
• Maintain clear documentation and support runbooks.

Required Skills

• Strong hands-on experience with Databricks (notebooks, jobs, workflows, clusters).
• Proficiency in PySpark, Python, and advanced SQL.
• Solid understanding of Delta Lake, data modeling, and ETL/ELT patterns.
• Experience supporting production workloads, with strong debugging and data validation skills.
• Working knowledge of Git and CI/CD, ideally Azure DevOps (Repos, Pipelines, Boards).