Databricks Developer

Tredence Inc.; San Jose (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Tredence Inc. is seeking a Databricks Developer to design, develop, and optimize scalable data pipelines on a Databricks Lakehouse platform. The role focuses on PySpark, Python, SQL, Delta Lake, and enterprise data modeling to support analytics, reporting, and AI/ML initiatives.

Responsibilities include building ETL/ELT processes, ensuring data quality and governance, and collaborating with stakeholders to deliver robust data solutions in a modern cloud environment.

Qualifications

• 4-8 years in data engineering and analytics.
• 3+ years hands-on Databricks and PySpark experience.
• Strong SQL development and data modeling skills.
• Experience implementing data quality and governance controls.
• Familiar with ETL/ELT pipelines and cloud data platforms.

Responsibilities

• Design and build scalable data pipelines on Databricks Lakehouse.
• Implement Delta Lake features: ACID, time travel, schema evolution.
• Collaborate with stakeholders to translate requirements into technical solutions.
• Optimize workloads for performance, scalability and cost.
• Govern data quality, lineage, metadata management and governance processes.
• Support cloud modernization and data platform initiatives.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

Data Modeling

Data Quality

Data Governance

ETL/ELT

Data Pipelines

Education

Bachelor's or Master’s degree in Computer Science / Information Technology / Data Engineering

Tools

Databricks

Unity Catalog

Data Catalog

Azure Databricks

Azure Data Factory

Azure Data Lake Storage

Azure Synapse Analytics

CI/CD (Azure DevOps, GitHub Actions)

GitHub Actions

Job description

We are looking for a highly skilled Databricks Developer with expertise in building and managing modern data platforms using the Databricks Lakehouse architecture. The ideal candidate will have strong experience in PySpark, Python, SQL, Delta Lake, data modeling, data quality frameworks, and enterprise-scale data engineering solutions. The role involves designing, developing, and optimizing scalable data pipelines that support analytics, reporting, and AI/ML initiatives.

Role description

Job Summary

Location -US/Canada

We are looking for a highly skilled Databricks Developer with expertise in building and managing modern data platforms using the Databricks Lakehouse architecture. The ideal candidate will have strong experience in PySpark, Python, SQL, Delta Lake, data modeling, data quality frameworks, and enterprise-scale data engineering solutions. The role involves designing, developing, and optimizing scalable data pipelines that support analytics, reporting, and AI/ML initiatives.

Key Responsibilities

• Design and implement scalable data solutions using Databricks Lakehouse Architecture.
• Develop and maintain data pipelines using PySpark, Python, and SQL.
• Build and optimize ETL/ELT workflows for batch and near real-time data processing.
• Implement Delta Lake features including ACID transactions, time travel, schema evolution, and data versioning.
• Design and maintain enterprise data models to support reporting and analytics requirements.
• Ensure data quality through validation, monitoring, reconciliation, and governance controls.
• Develop and manage data catalogs, metadata management, and data lineage processes.
• Collaborate with business stakeholders, architects, and analytics teams to gather and translate requirements into technical solutions.
• Optimize Databricks workloads for performance, scalability, and cost efficiency.
• Implement security, access controls, and governance best practices within the Databricks ecosystem.
• Support troubleshooting, root cause analysis, and production issue resolution.
• Contribute to data platform modernization and cloud migration initiatives.

Required Technical Skills

Databricks

• Strong experience with Databricks Architecture and platform administration.
• Hands-on expertise in Databricks Lakehouse Architecture.
• Deep understanding of Delta Lake concepts and implementation.
• Experience with Unity Catalog / Data Catalog and metadata management.
• Knowledge of Databricks Workflows, Jobs, Clusters, and Performance Tuning.

Data Engineering

• Strong proficiency in PySpark for large-scale data processing.
• Advanced Python programming skills.
• Expert-level SQL development and query optimization.
• Experience in building robust ETL/ELT pipelines.
• Strong understanding of data modeling techniques including:
  • Star Schema
  • Snowflake Schema
  • Dimensional Modeling
  • Data Vault (preferred)

Data Governance & Quality

• Experience implementing data quality frameworks and validation checks.
• Knowledge of data lineage, metadata management, and governance processes.
• Experience with data reconciliation, profiling, and monitoring tools.

Cloud & Platform Experience (Preferred)

• Azure Databricks
• Azure Data Lake Storage (ADLS)
• Azure Data Factory
• Azure Synapse Analytics
• CI/CD pipelines (Azure DevOps, GitHub Actions)

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.
• 4-8 years of experience in Data Engineering and Analytics.
• Minimum 3+ years of hands-on experience with Databricks and PySpark.
• Experience working in Agile development environments.

Preferred Certifications

• Databricks Certified Data Engineer Associate/Professional
• Microsoft Azure Data Engineer Associate (DP-203)
• Databricks Lakehouse Fundamentals

Key Deliverables

• Scalable and optimized data pipelines.
• Enterprise-grade Lakehouse solutions.
• High-quality curated datasets for analytics and reporting.
• Automated data quality and monitoring frameworks.
• Well-documented data models and metadata repositories.