PySpark / Databricks SQL consultant (contract)

Ted Calvert Consulting, LLC; Sacramento (CA); Remote; USD 96,000 - 165,000; Full time

Benefits offered by this job

Remote work

Occasional on-site meetings in SACRAM,

Job summary

Ted Calvert Consulting, LLC seeks an independent PySpark / Databricks SQL consultant for a contract engagement. You will support a State of California department on an Azure Databricks data lake, maintaining ingestion notebooks and resolving job failures.

Ideal candidates have 3–5 years of PySpark/Databricks experience, Unity Catalog, Medallion architecture, and strong Python skills, with PHI/PII data handling expertise. Mostly remote with some on-site Sacramento meetings; 1099/LLC preferred.

Qualifications

• 3-5 years of hands-on PySpark and Databricks development.
• Experience with Databricks Unity Catalog and Medallion architecture.
• Databricks SQL and strong Python skills.
• Experience handling PHI/PII data.

Responsibilities

• Maintain and enhance existing data ingestion notebooks.
• Troubleshoot and resolve failed jobs.
• Participate in code reviews.
• Document code and processes.
• Coordinate with the senior architect and team on priorities, standards, and code reviews.

Skills

PySpark

Databricks SQL

Python

PHI/PII handling

Code reviews

Tools

Azure Databricks

Unity Catalog

Medallion architecture

GitHub workflows

Job description

Seeking an independent PySpark / Databricks SQL consultant (contract)

We’re a small business supporting a State of California department on an Azure Databricks data lake, and we’re looking to add one experienced independent contractor to our crack contractor team.

The work:

• Maintain and enhance existing data ingestion notebooks
• Troubleshoot and resolve failed jobs
• Participate in code reviews
• Document code and processes
• Coordinate with the senior architect and team on priorities, standards, and code reviews

What we’re looking for:

• 3-5 years of hands-on PySpark and Databricks development
• Databricks Unity Catalog
• Medallion architecture in a data lake environment
• Databricks SQL
• Strong Python and PySpark skills
• Experience handling Protected Health Information (PHI) / Personally Identifiable Information (PII) data

Nice to have:

• Prior work with State of California departments
• GitHub (repo commands and workflow)
• Independent contractor (1099). We prefer candidates with their own LLC or business entity.
• Background check and data-security requirements may apply, given the nature of the data.
• Prefer candidate based in the Sacramento, California area but other California locations may be possible.
• Primarily remote work with occasional on-site meetings in Sacramento, California.
• Full time for up to three years.