Lead Data Engineer

Norfolk Southern Corp; Atlanta (GA); Hybrid; USD 150,000 - 230,000; Full time

Job summary

Norfolk Southern Corp seeks an experienced Lead Enterprise Data Engineer in Atlanta, GA. You will set technical direction for a Databricks-based lakehouse and streaming analytics platform, modernize ingestion and transformation patterns, and guide operations while collaborating with cross-functional partners to deliver production-grade data pipelines.

The role emphasizes Databricks Lakehouse expertise, streaming, data modeling, governance, and technical leadership to deliver scalable data

Qualifications

• Minimum of 10+ years in data engineering with Databricks Lakehouse experience.
• Strong hands-on with Spark/PySpark, Delta Lake, and Delta Live Tables; governance with Unity Catalog.
• Leadership in delivering end-to-end ETL/ELT pipelines in lakehouse environments.
• Experience with cloud analytics (AWS) and data security/privacy basics.
• Excellent SQL skills and data modeling capabilities across conceptual/logical/physical models.

Responsibilities

• Define data architecture direction for lakehouse, ingestion, and transformation patterns.
• Remove blockers via hands-on problem solving and architectural decisions across teams.
• Design enterprise data models across domains aligned with pipeline architecture.
• Develop and enforce data modeling standards and governance using Erwin and related tools.
• Ingest, wrangle, and validate large-scale structured and unstructured data for analytics.

Skills

Technical leadership

Cross-functional collaboration

Problem solving

Data modeling

SQL proficiency

Education

Bachelor’s degree in Information Systems/Computer Science/Computer Information Systems or related field

Tools

Databricks Lakehouse

Spark/PySpark

Delta Lake

Delta Live Tables

Unity Catalog

Genie

LakeFlow Designer

Kafka

Confluent

AWS analytics services (S3, IAM, Glue, Lambda, MSK)

SQL

Erwin

Python

Scala

Snowflake

BigQuery

HBASE

Cassandra

Azure

Kinesis

TIBCO EMS

IBM MQ Series

MSK

Git

REST API

Web Services

NoSQL

Job description

Lead enterprise data engineering for a Databricks-based lakehouse and streaming analytics platform at Norfolk Southern Corp. In this hybrid role in Atlanta, GA, you will set technical direction, modernize ingestion and transformation patterns, and help guide steady-state operations while collaborating with cross-functional partners to deliver production-grade data pipelines.

What you’ll do

• Lead the definition of data architecture with team members and stakeholders, establishing technical direction for lakehouse, ingestion, and transformation patterns.
• Identify and remove technical blockers by using hands-on problem solving, architectural decisions, and coordination across teams.
• Design and develop enterprise data models (conceptual, logical, and physical) across domains, ensuring alignment with lakehouse and pipeline architecture.
• Define and enforce data modeling standards, best practices, and governance processes using tools such as Erwin.
• Define data requirements, then gather, wrangle, and validate large-scale structured and unstructured data using data tools in the data environment.
• Enable data standardization, customization, and ad-hoc analysis by building mechanisms to ingest, analyze, validate, normalize, and clean data.
• Create data policy and develop interfaces and retention models that may require synthesizing or anonymizing data.
• Implement statistical data quality procedures for new data sources, using iterative analytics to support data scientists and analytics teams preparing data for visualization and insight generation.
• Develop and maintain data engineering best practices, contributing to insights related to data analytics and visualization.
• Lead a data engineering team to use state-of-the-art big data tools to build scalable architectures and high-performance pipelines for both real-time and batch data.
• Partner with data science and business intelligence teams to deliver models and pipelines for research, reporting, and machine learning.
• Build pipelines that clean, transform, and aggregate data from disparate sources.
• Use multiple languages and tools (for example, scripting) to integrate systems, and apply data architecture components from requirements through implementation while leading project teams.

What you bring

• Databricks Lakehouse expertise (Required): 10+ years in data engineering with deep hands-on experience across the Databricks ecosystem including Spark/PySpark pipelines, Delta Lake (merges, schema evolution), Delta Live Tables, Unity Catalog governance, Genie for self-service analytics, and LakeFlow Designer for visual ETL orchestration.
• Streaming and cloud infrastructure: 4+ years building reliable batch and streaming Spark workloads; 3+ years with Kafka (or Confluent) for high-volume event processing; and 3+ years using AWS analytics services including S3, IAM, Glue/Lambda/MSK.
• Data modeling and SQL: ability to build conceptual, logical, and physical data models with strong governance, along with advanced SQL skills for business-ready datasets.
• Technical leadership: experience leading technical design, mentoring engineers, building architectural consensus, and unblocking complex data engineering challenges.
• Delivery and collaboration: experience delivering ETL/ELT pipelines end-to-end in a lakehouse environment, working within Agile frameworks such as Scrum/Kanban/SAFe to manage iterative delivery and cross-team dependencies.

Tools and technologies

• Databricks, Spark, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, Genie, LakeFlow Designer
• Kafka, Confluent
• AWS analytics services (S3, IAM, Glue/Lambda/MSK)
• SQL, Erwin, Python, Scala
• Also listed: Snowflake, BigQuery, HBASE, Cassandra, Azure, Kinesis, TIBCO EMS, IBM MQ Series, MSK, Git, REST API, Web Services, NoSQL, ETL/ELT, and Agile frameworks (Scrum/Kanban/SAFe)

Work setup

• Location: Atlanta, GA, US
• Schedule: Hybrid (two days in office, three days remote)
• Shift work: No
• On-call: Yes
• Weekend work: No

Minimum education

Bachelor’s Degree, preferably in Information Systems, Computer Science, Computer Information Systems, or a related technology field.