Senior Databricks Engineer (Platform & Data Engineering)

Geisinger; United States; Hybrid; USD 90,000 - 130,000; Full time

Job summary

Geisinger is seeking a Databricks Platform Administrator/Data Engineer to architect innovative solutions and ensure data platform excellence. This role involves managing Databricks and building data pipelines using PySpark and T-SQL.

Ideal candidates will have strong skills in Python and automation, and a commitment to data governance. With a focus on collaboration, this position offers the flexibility of work from home while contributing to exciting data initiatives.

Qualifications

• Solid experience with Databricks administration, including workspace management.
• Strong proficiency in Python and scripting with Bash and PowerShell.
• Experience with PySpark and/or T-SQL for data processing.

Responsibilities

• Manage workspaces, clusters, jobs, and overall platform health.
• Build and optimize data pipelines using PySpark and T-SQL.
• Create automation and tooling using Python, Bash, and PowerShell.

Skills

Databricks administration

Python programming

Bash scripting

PowerShell scripting

PySpark

T-SQL

Data governance

Problem-solving

Education

High School Diploma or Equivalent (GED)

Tools

Azure DevOps

Terraform

AWS

MS SQL Server

Job description

Location: Work from home (Pennsylvania) Shift: Days (United States of America) Scheduled Weekly Hours: 40 Worker Type: Regular Exemption Status: Yes

Job Summary

We are seeking a Databricks Platform Administrator / Data Engineer who enjoys development, application configuration, and building innovative solutions. This role is ideal for someone who thrives on possibilities, sees the bigger picture, and consistently delivers the best version of any task. You will architect solutions from conversational ideas, break work into actionable tasks in Microsoft Planner or Azure DevOps, and collaborate across a dynamic team to drive data platform excellence. You'll be responsible for the performance of tasks related to application implementation, adoption, enhancement, and support. This includes but is not limited to analysis, development, training, testing, documentation, go-live and ongoing support, problem resolution, and maintenance.

Key Responsibilities

• Databricks Administration: Manage workspaces, clusters, jobs, security configurations, libraries, and overall platform health.
• Data Engineering: Build and optimize data pipelines and transformations using PySpark and T-SQL.
• Automation & Development: Create automation and tooling using Python, Bash, and PowerShell.
• Governance & Security: Implement and automate data governance, access controls, auditability, and compliance best practices. Note: A significant portion of our development focuses on governance and security automation.
• Solution Architecture: Translate conversational ideas into clear solution architectures; break work into tasks in Planner or Azure DevOps for team execution.
• Effort Estimation: Estimate level of effort and forecast delivery dates with reasonable accuracy.
• Documentation: Produce high-quality documentation including workflows, gap analyses, design specs, build docs, test plans, and transition documentation.
• Testing Oversight: Oversee unit, scenario-based, integrated, capacity, and parallel testing.
• On-Call Support: Participate in a 24x7x365 on-call rotation to support Databricks, SQL Server, and other relevant applications supported by the team.
• Upgrades & Monitoring: Support system upgrades, patches, performance tuning, observability, and proactive monitoring.
• Data Integrity: Maintain accountability for data integrity across applications and pipelines.
• Change Management: Complete system build and documentation in adherence to departmental guidelines and change management policy.
• Peer Review: Review teammates’ work and provide constructive feedback.
• Communication & Escalation: Communicate effectively with peers, stakeholders, and leadership; provide regular status updates and elevate issues appropriately.
• Facilitation & Leadership: Drive discussions, mentor accountable leaders, and coordinate resources across initiatives.
• Training & Mentorship: Provide feedback on training materials; conduct training sessions; develop and mentor other team members.
• Policies & Vendor Engagement: Help develop department guidelines, standards, procedures, and policies; facilitate vendor reviews and selection processes.
• Integrations (as applicable): Demonstrate understanding of integration and technical requirements for peripherals/devices that interface with assigned applications.
• Support Coordination: Coordinate remote and/or on-site support as needed for end-users.

Optional Contributions

• Create complex AI/BI dashboards.
• Contribute to CI/CD pipelines and integrations using Azure DevOps or GitHub.
• Apply Terraform for infrastructure as code.
• Provide MS SQL Server administrative support.

Required Skills & Qualifications

• Solid experience with Databricks administration, including workspace management, cluster configuration, security, jobs, and libraries.
• Strong proficiency in Python and scripting with Bash and PowerShell.
• Experience with PySpark and/or T-SQL for data processing and transformation.
• Excellent communication skills with the ability to articulate technical concepts clearly to varied audiences.
• Demonstrated critical thinking, problem-solving, and troubleshooting skills.
• Ability to estimate effort and deliver on time.
• Strong understanding of data governance principles and best practices.
• Commitment to information security best practices.

Preferred Qualifications

• Familiarity with Terraform for infrastructure as code.
• Experience creating AI/BI dashboards.
• Knowledge of CI/CD pipelines and integration using Azure DevOps or GitHub.
• Hands‑on experience with AWS data ingestion tools (e.g., Glue, Lambda).
• Experience with MS SQL Server administration.
• Knowledge of cloud architecture and multi‑cloud environments.

Education

High School Diploma or Equivalent (GED) - Required.

Experience

Minimum of 3 years Managing people, processes, or projects (Required). Minimum of 12 years Relevant experience* (Required).

Certification(s) and License(s)

Skills: Communication, Critical Thinking, Interpersonal Communication.

We are an affirmative action, equal opportunity employer and all qualified applicants will receive consideration for employment regardless to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran.