Lead Data Engineer - Databricks & AI Platform (Remote)

pennfostercarrus; United States; Remote; USD 150,000 - 200,000; Full time

Job summary

Penn Foster Group seeks a Lead Data Engineer to shape our cloud data platform and AI strategy. You will lead a Databricks Lakehouse initiative, mentor engineers, and establish engineering standards across ingestion, transformation, governance, and analytics.

You will partner with BI, Product, Platform Engineering, Security, and MLOps teams to build scalable, governed data products enabling self-service analytics and AI-powered decisions in a remote US-based role.

Qualifications

• Bachelors degree in CS, Engineering, Information Systems, or related field.
• 8+ years of Data Engineering/Analytics engineering experience.
• 3+ years leading technical initiatives and mentoring teams.
• Strong communication and collaboration with stakeholders.
• Experience with Databricks Lakehouse and Azure-based data platforms.

Responsibilities

• Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices.
• Mentor and develop junior/mid-level Data Engineers and set engineering standards.
• Design scalable data pipelines, governance models, and observability for reliability and cost efficiency.
• Collaborate with BI, Product, Platform Engineering, Security, Data Governance, and MLOps teams to deliver scalable data products.
• Drive modernization initiatives and adoption of Databricks capabilities and AI-enabled data platforms.

Skills

Databricks Lakehouse

SQL

Python

Leadership

Mentoring

Education

Bachelor's degree in CS/Engineering or related field

Tools

Databricks

Azure

Genie

Unity Catalog

Job description

Penn Foster Group seeks a Lead Data Engineer to shape our cloud data platform and AI strategy. You will lead a Databricks Lakehouse initiative, mentor engineers, and establish engineering standards across ingestion, transformation, governance, and analytics.

You will partner with BI, Product, Platform Engineering, Security, and MLOps teams to build scalable, governed data products enabling self-service analytics and AI-powered decisions in a remote US-based role.