Manager Data Engineer

pennfostercarrus; United States; Remote; USD 150,000 - 200,000; Full time

Job summary

Penn Foster Group seeks a Lead Data Engineer to shape our cloud data platform and AI strategy. You will lead a Databricks Lakehouse initiative, mentor engineers, and establish engineering standards across ingestion, transformation, governance, and analytics.

You will partner with BI, Product, Platform Engineering, Security, and MLOps teams to build scalable, governed data products enabling self-service analytics and AI-powered decisions in a remote US-based role.

Qualifications

• Bachelors degree in CS, Engineering, Information Systems, or related field.
• 8+ years of Data Engineering/Analytics engineering experience.
• 3+ years leading technical initiatives and mentoring teams.
• Strong communication and collaboration with stakeholders.
• Experience with Databricks Lakehouse and Azure-based data platforms.

Responsibilities

• Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices.
• Mentor and develop junior/mid-level Data Engineers and set engineering standards.
• Design scalable data pipelines, governance models, and observability for reliability and cost efficiency.
• Collaborate with BI, Product, Platform Engineering, Security, Data Governance, and MLOps teams to deliver scalable data products.
• Drive modernization initiatives and adoption of Databricks capabilities and AI-enabled data platforms.

Skills

Databricks Lakehouse

SQL

Python

Leadership

Mentoring

Education

Bachelor's degree in CS/Engineering or related field

Tools

Databricks

Azure

Genie

Unity Catalog

Job description

Manger Data Engineer

Department: Data & Analytics Reports To: Director, Data Engineering Location: Remote (U.S.)

Position Summary

Penn Foster Group is seeking an experienced Lead Data Engineer to help shape the future of our enterprise data platform and AI strategy. This is a highly visible technical leadership role responsible for architecting, building, and evolving our modern cloud data platform while establishing engineering standards, mentoring a growing team, and driving innovation across the organization.

As a Lead Data Engineer, you will serve as the technical leader for our Databricks Lakehouse platform, providing hands‑on leadership across data ingestion, transformation, governance, analytics enablement, and AI‑ready data products. You will partner closely with Business Intelligence, Product, Platform Engineering, Security, Data Governance, and MLOps teams to build trusted, scalable, and governed data solutions that power enterprise analytics, machine learning, and generative AI.

This role combines deep technical expertise with leadership, mentoring, and strategic influence. You'll help establish the long‑term technical direction of Penn Foster Group's data ecosystem while coaching and developing engineers, driving engineering excellence, and implementing modern data engineering best practices.

Why Join Penn Foster Group?

Penn Foster Group is in the midst of an exciting enterprise-wide data platform modernization journey. We are investing in a modern cloud‑native architecture built on Databricks, Azure, and AI‑enabled technologies that will become the foundation for analytics, operational reporting, machine learning, and generative AI across the company.

This is a unique opportunity to join at a transformational point in our technology journey. Rather than simply maintaining existing systems, you'll help define the future of our data platform—shaping architecture, engineering standards, governance, and the adoption of emerging AI technologies that will influence how data is used across the organization for years to come. You will also be a key player in our drive for Self Service Analytics at PFG.

As our Lead Data Engineer, you will have the opportunity to:

• Help define the long‑term strategy for Penn Foster Group's modern Data & AI platform.
• Influence enterprise architecture decisions around Databricks, Azure, Unity Catalog, semantic data products, and AI‑powered analytics.
• Build modern, scalable data products that enable self‑service analytics and generative AI.
• Lead adoption of emerging Databricks capabilities, including Genie, AI/BI, Unity Catalog, Lakeflow, and other evolving platform innovations.
• Mentor and develop a growing Data Engineering team while establishing engineering standards that scale across the organization.
• Partner directly with technology and business leaders to solve meaningful business problems using modern cloud and AI technologies.

Key Responsibilities

Technical Leadership

• Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices.
• Mentor and develop junior and mid-level Data Engineers, fostering technical growth, knowledge sharing, and continuous learning.
• Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence.
• Lead technical design discussions, evaluate architectural tradeoffs, and drive adoption of modern engineering practices.
• Champion a culture of quality, collaboration, innovation, and continuous improvement.

Data Platform & Databricks Leadership

• Serve as the technical lead for Penn Foster Group's Databricks Lakehouse platform.
• Design, build, and support scalable enterprise data pipelines using SQL, Python, Apache Spark, and Databricks.
• Define best practices for Databricks development, including Workflows, Repos, Jobs, notebooks, reusable Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation.
• Design and optimize Delta Lake architectures using Medallion patterns, Delta optimization, Liquid Clustering, and Photon.
• Build reusable ingestion frameworks supporting batch, streaming, CDC, and API‑based integration patterns.
• Optimize Spark workloads for performance, scalability, reliability, and cloud cost efficiency.

AI & Databricks Genie Enablement

• Partner with Analytics and business stakeholders to develop trusted semantic data products that power Databricks Genie.
• Design and maintain Genie Spaces and Genie Ontologies that accurately represent business entities, relationships, metrics, and terminology.
• Establish best practices for semantic modeling, governed metrics, business metadata, and AI‑ready datasets.
• Evaluate and implement emerging Databricks AI capabilities to improve self‑service analytics and business productivity.
• Collaborate with Data Science and MLOps teams to enable machine learning, generative AI, and advanced analytics initiatives.

Data Architecture, Governance & Operational Excellence

• Design scalable Lakehouse architectures supporting analytics, reporting, AI, and operational data products.
• Partner with Data Governance teams to implement enterprise governance using Unity Catalog, including data lineage, fine‑grained security, metadata management, and access controls.
• Champion automated testing, monitoring, observability, data quality, and production reliability.
• Serve as the technical escalation point for complex production issues and lead root cause analysis and continuous improvement efforts.
• Drive platform modernization initiatives while balancing delivery, scalability, maintainability, and operational excellence.

Cross‑Functional Partnership

• Collaborate with Business Intelligence, Product, Platform Engineering, Security, and MLOps teams to deliver trusted, scalable data products.
• Translate business requirements into modern technical solutions that support enterprise reporting, analytics, AI, and strategic decision‑making.
• Contribute to technical roadmaps, platform strategy, and the continued evolution of Penn Foster Group's Data & Analytics capabilities.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.
• 8+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering.
• 3+ years of experience leading technical initiatives and mentoring engineering teams.
• Demonstrated success developing engineers while driving engineering excellence and technical standards.
• Strong communication and collaboration skills with the ability to influence both technical and business stakeholders.

Technical Qualifications

Required

• Expert knowledge of the Databricks Lakehouse Platform, including:
• Apache Spark (PySpark)
• Delta Lake
• Unity Catalog
• Databricks Workflows & Jobs
• Repos
• SQL Warehouses
• Cluster Policies
• Serverless Compute
• MLflow
• Lakehouse Monitoring
• Auto Loader
• Delta Live Tables / Lakeflow

Expert Knowledge in SQL and Python.

Strong Experience working with BI tools such as PowerBI, Tableau or Business Objects

Strong Microsoft Azure experience, including ADLS Gen2, Microsoft Entra ID (Azure AD), RBAC, networking, and cloud security.

Experience implementing Medallion Architecture, dimensional modeling, and domain-oriented data products.

Deep understanding of Spark optimization techniques, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization.

Experience implementing CI/CD pipelines, Git-based development workflows, Infrastructure as Code, and automated testing.

Strong understanding of data governance, metadata management, data quality, observability, security, and compliance.

Preferred

• Experience implementing Databricks Genie, Genie Spaces, and Genie Ontologies.
• Experience designing semantic models and AI‑ready data products.
• Experience supporting Power BI, Tableau, or other enterprise BI platforms.
• Experience with machine learning platforms, MLOps, or Generative AI applications.
• Experience with dbt, Great Expectations, or similar modern data engineering tools.
• Databricks Certified Data Engineer Professional and/or Microsoft Azure certifications.

Working Conditions & Hiring Process:

U.S. Work Location Requirement: All employees must reside in and perform work within the United States. We do not support international remote work arrangements for this role.

Travel Expectations: While this is a remote position, occasional travel may be required for team meetings, company events, conferences, summits, etc. Candidates must be willing and able to travel as needed.

On‑Camera Work Environment: We operate in a highly collaborative, remote, on‑camera culture. Employees are expected to be on camera during meetings unless extenuating circumstances apply.

Recorded Video Interviews: All video interviews conducted during the hiring process will be recorded to support internal hiring consistency and process improvement.

Background Checks: Employment with Penn Foster Group is contingent upon successfully completing applicable pre‑employment screening requirements, which may include verification of employment history, education, and criminal background, where permitted by law.

Identity Verification (Form I-9): As part of our onboarding process, all hires must complete employment eligibility verification in compliance with federal law. This includes remote Form I-9 verification, which may require an in‑person identity verification step with an authorized representative.

About Us:

At Penn Foster Group, we are transforming online learning to help learners by uniting Penn Foster, CareerStep, Ashworth College, James Madison High School, the New York Institute of Photography, the New York Institute of Art and Design, and other education platforms. Together, we create an accelerated path to greater economic mobility through real‑world skills and knowledge that enable them to achieve long‑term success in the workplaces of the future. Our history dates back to 1890 when our founder, Thomas Foster, pioneered distance education by offering training by mail for coal miners to get the necessary skills for safer jobs. Today, with the partners who use our education and training programs, we continue that mission of providing accessible training and education for in‑demand skills and are building a workforce that’s prepared for the future job market.

Equal Employment Opportunity:

We strive toward Diversity, Equity, and Inclusion at Penn Foster Group by intentionally building teams that are diverse – in identities, lived experiences, and ideas to create a culture where people feel connected to each other and have a sense of belonging. We value diversity, equity, and inclusion because it is the foundation that enables us to achieve what we set out to do as an organization – from maximizing the number of learners who can reach their goals while giving them the kinds of experiences we want them to have, to becoming the type of company we want to work in.

What We Offer:

We offer a robust benefits package that includes medical, dental, vision, flexible spending, generous paid time off, sponsored volunteer opportunities, a 401K with a company match, plus free access to all of our online programs.