Azure Databricks Data Engineer Unity Catalog and Governance

System One; Pittsburgh (Allegheny County); On-site; USD 110,000 - 165,000; Full time

Job summary

System One is seeking a Data Engineer - Databricks for a contract-to-hire role in Pittsburgh, onsite. You will help build modern, cloud-native data platforms on Microsoft Azure, with a focus on Unity Catalog, governance, and enterprise analytics in the Investment/Capital Markets domain.

The ideal candidate designs scalable data pipelines using Azure Databricks, PySpark, SQL, and Delta Lake, and collaborates with architects and analysts in an Agile environment to deliver secure, high-performance

Qualifications

• Minimum 5 years of hands-on experience with Azure Databricks and Spark.
• Strong knowledge of Unity Catalog and data governance.
• Experience with Delta Lake, ETL/ELT development, and Azure data services.
• Proficient in Python (PySpark) and SQL.
• Experience with Git & CI/CD practices in Agile environments.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Build and optimize ETL/ELT workflows with PySpark, SQL, and Delta Lake.
• Implement Unity Catalog to support enterprise data governance and security.
• Develop cloud-native data engineering solutions on Azure.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Ensure data quality, reliability, and compliance through monitoring and best practices.
• Support deployments, troubleshooting, and performance tuning.
• Participate in code reviews and technical documentation.
• Apply DevOps practices and cloud security principles in Azure environments.

Tools

Azure Databricks

Unity Catalog

Delta Lake

Azure Data Factory

ADLS Gen2

Git

CI/CD

Terraform/Bicep

Job description

System One is seeking a Data Engineer - Databricks for a contract-to-hire role in Pittsburgh, onsite. You will help build modern, cloud-native data platforms on Microsoft Azure, with a focus on Unity Catalog, governance, and enterprise analytics in the Investment/Capital Markets domain.

The ideal candidate designs scalable data pipelines using Azure Databricks, PySpark, SQL, and Delta Lake, and collaborates with architects and analysts in an Agile environment to deliver secure, high-performance