Lead Data Engineer: Databricks Lakehouse & AI (Remote)

Penn Foster Group; Northern (KY); Hybrid; USD 150,000 - 190,000; Full time

Job summary

Penn Foster Group is seeking an experienced Lead Data Engineer to shape our enterprise data platform and AI strategy. This role focuses on architecting, building, and evolving a modern cloud data platform, mentoring a growing team, and driving engineering excellence across governance, analytics, and AI-enabled data products.

As technical lead for the Databricks Lakehouse, you will collaborate with BI, Product, Platform Engineering, Security, Data Governance, and MLOps teams to deliver scalable,

Qualifications

• 8+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering.
• 3+ years of experience leading technical initiatives and mentoring engineering teams.
• Strong communication and collaboration skills with the ability to influence both technical and business stakeholders.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.

Responsibilities

• Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices.
• Mentor and develop junior and mid-level Data Engineers, fostering technical growth, knowledge sharing, and continuous learning.
• Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence.
• Lead technical design discussions, evaluate architectural tradeoffs, and drive adoption of modern engineering practices.
• Collaborate with Business Intelligence, Product, Platform Engineering, Security, and MLOps teams to deliver trusted, scalable data products.
• Design scalable Lakehouse architectures and governance practices with Unity Catalog.

Skills

Databricks Lakehouse

PySpark

SQL

Python

Unity Catalog

Delta Lake

Genie

MLOps

Power BI

Education

Bachelor's degree in CS/Engineering/IS

Tools

Azure

Git

Job description

Penn Foster Group is seeking an experienced Lead Data Engineer to shape our enterprise data platform and AI strategy. This role focuses on architecting, building, and evolving a modern cloud data platform, mentoring a growing team, and driving engineering excellence across governance, analytics, and AI-enabled data products.

As technical lead for the Databricks Lakehouse, you will collaborate with BI, Product, Platform Engineering, Security, Data Governance, and MLOps teams to deliver scalable,