Senior Azure Databricks Platform Engineer

Insight Global; Austin (TX); On-site; USD 150,000 - 230,000; Full time

Job summary

Insight Global is seeking a Senior Data Platform Engineer to design, develop, and support an enterprise cloud data platform. You will lead development on Azure Databricks, Delta Lake, and related tooling, while owning operations, security, and cost optimization across production workloads.

You will mentor engineers, drive data quality, and implement scalable data products, with a focus on reliability, governance, and platform modernization.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, Data Science, or a related field.
• Master's degree preferred.
• 10+ years of experience in data engineering, data warehousing, or data platform engineering.
• 5+ years of experience designing or implementing cloud-based data warehouse, data lake, or data platform solutions.
• 3+ years of hands-on Azure Databricks experience in enterprise production environments.
• Experience with ETL/ELT development, data modeling, large-scale integration, and cloud platform modernization.
• Experience leading technical implementations, solution design, design reviews, and production releases.
• Experience supporting production environments, incident response, root cause analysis, and operational support processes.
• Experience in regulated, financial services, banking, or audit-sensitive environments preferred.

Responsibilities

• Design, develop, and maintain scalable data pipelines, ingestion frameworks, transformation processes, and reusable data products using Azure Databricks, PySpark, SQL, and Delta Lake.
• Implement Bronze, Silver, and Gold architecture patterns supporting enterprise reporting, analytics, AI, and self-service data consumption.
• Design, build, and operationalize AI agent solutions on the Databricks platform, integrating governed enterprise data, models, tools, and APIs for secure and scalable business use cases.
• Build reusable frameworks, utilities, and platform components that improve engineering productivity, quality, consistency, and deployment speed.
• Develop and support batch, near-real-time, and streaming integration solutions, including modernization of legacy warehouse and ETL workloads.
• Serve as technical lead for complex initiatives; develop solution designs, lead technical reviews, recommend tools and approaches, and guide work through production implementation.
• Partner with data architects and platform leaders to ensure solutions are scalable, secure, governed, cost-conscious, and operationally supportable.
• Mentor engineers and promote standards for coding, testing, documentation, performance, and production readiness.
• Implement data quality, validation, reconciliation, monitoring, metadata, and lineage capabilities; support RBAC and enterprise security controls.
• Build and maintain CI/CD, automated testing, deployment, and release processes using Azure DevOps and Git-based practices.
• Contribute to platform observability, alerting, operational dashboards, health metrics, performance tuning, and cost optimization.
• Participate in production support, incident response, pager, and on-call rotations; troubleshoot issues, lead root cause analysis, and implement durable remediation.
• Create and maintain operational runbooks, support procedures, technical documentation, and knowledge-sharing assets.
• Collaborate with architecture, governance, security, analytics, application, and business teams to translate requirements into scalable platform solutions.
• Support technical discovery, estimation, roadmap planning, delivery execution, and evaluation of emerging cloud, data, and AI capabilities.

Skills

SQL

Python

PySpark

Spark performance optimization

Analytical troubleshooting

Mentoring

Communication

Leadership

Education

Bachelor's degree in Computer Science/Engineering/Data Science

Master's degree preferred

Tools

Azure Databricks

Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

MLflow

Agent Bricks/Agent Framework

Power BI

CI/CD

Azure DevOps

Git

Job description

Insight Global is seeking a Senior Data Platform Engineer to design, develop, and support an enterprise cloud data platform. You will lead development on Azure Databricks, Delta Lake, and related tooling, while owning operations, security, and cost optimization across production workloads.

You will mentor engineers, drive data quality, and implement scalable data products, with a focus on reliability, governance, and platform modernization.