Databricks Engineer

Princeton IT Services, Inc; United States; Remote; USD 130,000 - 180,000; Full time

Job summary

Princeton IT Services, Inc. is seeking a Databricks Engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake.

You will work on ingestion, transformation, and processing frameworks, with CI/CD automation and secure data governance. The ideal candidate has 6+ years of data engineering experience, 3+ years Databricks hands-on expertise, and a Bachelor's in Engineering.

Qualifications

• Bachelor's degree in Engineering or related field.
• 6+ years in data engineering with 3+ years Databricks hands-on experience.
• Strong Python, PySpark and SQL skills; Delta Lake expertise.

Responsibilities

• Design, develop, and maintain scalable data pipelines with Databricks, PySpark, Python, and SQL.
• Build batch and streaming ingestion frameworks for high-volume data.
• Develop reusable, metadata-driven ingestion and processing frameworks.
• Implement Delta Lake architectures for reliability and performance.
• Configure Unity Catalog access controls and data governance.
• Use Auto Loader for incremental file ingestion and Delta Live Tables/Lakeflow for transforms.
• Implement CI/CD workflows with Azure DevOps and Databricks Asset Bundles.
• Troubleshoot pipeline failures and ensure production reliability.
• Optimize Spark jobs and SQL queries for performance and cost.
• Collaborate with cross-functional teams to deliver reliable data solutions.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

Data Engineering

Unity Catalog

Auto Loader

DLT/Lakeflow

Azure DevOps

Databricks Asset Bundles

Performance Optimization

Education

Bachelor's degree in Engineering

Tools

Azure DevOps

Databricks Asset Bundles

Delta Live Tables (DLT)/Lakeflow

Auto Loader

Job description

Job Title: Databricks Engineer

Work Location: Remote United States

Candidate Location: Must Currently Reside in the United States

Experience: 6+ years of overall experience, including 3+ years of hands-on Databricks experience

Industry Preference: Healthcare, preferably Health Payer

Education: Bachelor s degree in Engineering (minimum requirement)

Travel: No client-paid travel or expenses

Extension: Possible, depending on candidate performance

Python Proficiency: P3 Advanced

We are seeking an experienced Databricks Engineer to design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, SQL, and Delta Lake. The ideal candidate will have strong expertise in enterprise data ingestion, transformation, and processing, along with experience building reusable frameworks for high-volume and metadata-driven data pipelines. The candidate will work with modern Databricks capabilities, including Unity Catalog, Auto Loader, Delta Live Tables (DLT)/Lakeflow, and CI/CD automation using Azure DevOps and Databricks Asset Bundles. Experience in the healthcare payer industry is highly preferred.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
• Build robust batch and streaming ingestion frameworks to process large volumes of enterprise data.
• Develop reusable, metadata-driven frameworks for data ingestion, transformation, and processing.
• Implement and optimize Delta Lake architectures to ensure data reliability, scalability, and performance.
• Configure and manage data governance, access controls, and security using Unity Catalog.
• Implement incremental data ingestion and automated file processing using Auto Loader.
• Develop and maintain Delta Live Tables (DLT)/Lakeflow pipelines for reliable data transformation and processing.
• Implement data-quality checks, validation rules, error handling, and logging mechanisms.
• Optimize Spark jobs and SQL queries for performance, scalability, and efficient resource utilization.
• Implement CI/CD workflows and deployment automation using Azure DevOps and Databricks Asset Bundles.
• Collaborate with cross-functional teams to understand business requirements and deliver reliable data solutions.
• Troubleshoot pipeline failures, investigate data discrepancies, and improve overall pipeline reliability.
• Follow engineering best practices for code quality, documentation, testing, and production support.

Required Technical Skills

• Databricks: Strong hands-on experience developing and managing production-grade data pipelines.
• PySpark: Advanced proficiency in distributed data processing and Spark transformations.
• Python: Advanced programming skills, including reusable frameworks, modular code, and error handling.
• SQL: Strong knowledge of complex queries, joins, aggregations, and query optimization.
• Delta Lake: Experience with Delta tables, incremental processing, and reliable data architectures.
• Data Engineering: Expertise in batch and streaming ingestion, ETL/ELT processes, and data transformation.
• Unity Catalog: Understanding of data governance, access controls, and permissions.
• Auto Loader: Experience implementing scalable and incremental file ingestion.
• DLT/Lakeflow: Experience developing and managing data transformation pipelines.
• Azure DevOps: Experience with source control, CI/CD pipelines, and deployment automation.
• Databricks Asset Bundles: Experience packaging, configuring, and deploying Databricks projects across environments.
• Performance Optimization: Ability to troubleshoot and optimize Spark jobs, queries, and data pipelines.

Preferred Qualifications

• Experience working in the healthcare payer industry.
• Familiarity with healthcare data, payer operations, claims processing, member enrollment, eligibility, and provider data.
• Experience developing enterprise-scale data platforms and reusable ingestion frameworks.
• Knowledge of data governance, data quality, and secure data processing practices.
• Experience supporting production data pipelines in cloud-based environments.

Minimum Qualifications

• 6+ years of overall professional experience in data engineering or a related field.
• At least 3 years of hands-on Databricks development experience.
• Strong proficiency in Python, PySpark, and SQL.
• Demonstrated experience building scalable data pipelines using Delta Lake.
• Experience with batch and streaming ingestion patterns.
• Hands-on experience with modern Databricks tools and deployment workflows.
• Bachelor s degree in Engineering or a related engineering discipline.
• Must currently reside in the United States and be available to work remotely.