Lead Data Engineer (Databricks, Big Data)

Norfolk Southern; Atlanta (GA); Hybrid; USD 170,000 - 210,000; Full time

Job summary

Norfolk Southern invites a Lead Data Engineer to shape our Databricks lakehouse architecture and streaming analytics capabilities. You will own data pipelines, partner with data scientists, BI teams, and cross‑functional stakeholders to deliver scalable, production-ready solutions.

We seek a hands-on leader with 10+ years in data engineering, strong SQL, and cloud experience. This role is based in Atlanta with a hybrid in-office/remote schedule and growth opportunities within a long‑standing

Qualifications

• Databricks Lakehouse expertise with hands-on experience across Spark/PySpark pipelines.
• Experience building batch/streaming Spark workloads in cloud environments.
• Strong governance and data modeling practices with SQL proficiency.

Responsibilities

• Lead the definition of data architecture for lakehouse, ingestion, and transformation patterns.
• Unblock delivery by hands-on problem solving and cross-team coordination.
• Design and develop enterprise data models across domains to align with architecture.
• Define data modeling standards and governance using tools like Erwin.
• Gather and wrangle large-scale structured and unstructured data for analytics.
• Develop data policies and interfaces, focusing on data anonymization where needed.
• Drive data quality via statistical procedures and support data scientists and BI teams.
• Mentor engineers, lead architectural decisions, and ensure delivery quality.

Skills

Databricks Lakehouse

Streaming & Cloud

Data Modeling & SQL

Technical Leadership

Delivery & Collaboration

PySpark/Scala

Education

Bachelor's Degree

Tools

Databricks

Spark/PySpark

Delta Lake

Unity Catalog

Kafka

Job description

Requisition 40310: B5 Lead Data Engineer (Databricks, Big Data)

A resume helps you stand out to hiring managers and recruiters; your resume communicates your experience and your brand. While it is not required, we encourage you to include an up-to-date resume along with a completed job application to give you the best opportunity to be considered. A complete resume helps us to better understand your unique background, relevant experiences, and passions. We look forward to learning about you.

Norfolk Southern offers a unique opportunity to be part of our proud legacy that spans nearly 200 years. We are a customer‑centric, operations‑driven team dedicated to advancing safety, serving communities, and driving innovation for tomorrow's rail. As part of Norfolk Southern, you'll join a collaborative team where there are opportunities for growth across the organization. We are building a culture where everyone can thrive by owning and driving exceptional results, being humble and leading with trust, serving our customers with excellence, and collaborating and coaching to win.

Job Description

Norfolk Southern Corporation is seeking a Lead Data Engineer who enjoys collaborating across the organization to deliver reliable, scalable data solutions. Join a skilled team focused on modern data platforms, where you will lead the definition of architecture, help shape and operate our Databricks‑based lakehouse and streaming analytics capabilities, and guide the team through technical delivery.

This role blends strong business and analytical judgment with hands‑on engineering and technical leadership: you will partner with stakeholders and the engineering team to understand problems, translate them into clear requirements, define and drive pragmatic architectures, and deliver production‑grade ingestion and transformation pipelines - with ownership through deployment and steady‑state operations. You will also work closely with the team to proactively identify and remove technical blockers that stand in the way of delivery.

In this role, you will work with teams across the company to clarify Data and Analytics needs, perform structured requirements discovery alongside Data Modelers, align with BI development on scope and estimates, and lead and guide delivery through completion while promoting engineering quality, consistency, and best practices across the team.

Job Responsibilities

• Lead the definition of data architecture in partnership with the team and stakeholders, setting technical direction for the lakehouse, ingestion, and transformation patterns.


• Identify, accelerate, and remove technical blockers for the team - unblocking delivery through hands‑on problem solving, design decisions, or coordination with other teams.


• Lead the design and development of enterprise data models (conceptual, logical, and physical) across domains, ensuring they align with lakehouse and pipeline architecture.


• Define and enforce data modeling standards, best practices, and governance processes, leveraging tools such as Erwin.


• Define data requirements, gather, and wrangle large scale of structured and unstructured data, and validate data by running various data tools in the Data Environment.


• Support the standardization, customization and ad‑hoc data analysis, and will develop the mechanisms to ingest, analyze, validate, normalize and clean data.


• Create data policy and develop interfaces and retention models which require synthesizing or anonymizing data.


• Implement statistical data quality procedures on new data sources, and by applying rigorous iterative data analytics, supports Data Scientists and analytics and insights creation in data sourcing and preparation to visualize data and synthesize insights of commercial value.


• Develops and maintains data engineering best practices and contributes to Insights on data analytics and visualization concepts, methods and techniques.


• Lead Data Engineering team to use state‑of‑the‑art Big Data tools and technologies to build scalable data architecture and performant data pipelines to acquire high‑velocity real‑time and batch data of different sizes and scales


• Work closely with the data science and business intelligence teams to develop data models and pipelines for research, reporting, and machine learning.


• Build data pipelines that clean, transform, and aggregate data from disparate sources.


• Employ a variety of languages and tools (e.g. scripting languages) to marry systems together.


• Apply knowledge of Data Architecture components, leads project teams from requirements to implementation.

Education Required

Bachelor's Degree, preferably in Information Systems, Computer Science, Computer Information Systems or related technology field.

Skills Required

• Databricks Lakehouse Expertise (Required): 10+ years in data engineering with deep, hands‑on experience across the Databricks ecosystem including Spark/PySpark pipelines, Delta Lake (merges, schema evolution), Delta Live Tables, Unity Catalog governance, Genie for self‑service analytics, and LakeFlow Designer for visual ETL orchestration.


• Streaming & Cloud Infrastructure: 4+ years building reliable batch/streaming Spark workloads, 3+ years with Kafka (or Confluent) for high‑volume event processing, and 3+ years working with AWS analytics services (S3, IAM, Glue/Lambda/MSK).


• Data Modeling & SQL: Proven ability to design conceptual, logical, and physical data models with strong governance practices, paired with advanced SQL skills for building reliable, business‑ready datasets.


• Technical Leadership: Track record of leading technical design, mentoring engineers, driving architectural consensus, and unblocking teams on complex data engineering challenges.


• Delivery & Collaboration: Experience delivering ETL/ELT pipelines end‑to‑end in a lakehouse environment, working within Agile frameworks (Scrum/Kanban/SAFe) to manage iterative delivery and cross‑team dependencies.

Skills Desired

• Prior experience as a technical lead, staff engineer, or similar role with formal responsibility for architecture decisions and team unblocking.


• Python (and/or Scala) and PySpark/Scala‑Spark.


• Database solutions like Snowflake, Delta Lake or BigQuery.


• NoSQL databases, including HBASE and/or Cassandra.


• Azure, AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue.


• Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.


• Strong understanding of Relational & Dimensional modeling.


• Experience with GIT code versioning software.


• Experience with REST API and Web Services.

Work Conditions

Location: Atlanta, GA
Environment: Hybrid (two days in office; three days remote)
Shift Work: No
On-Call: Yes
Weekend Work: No

Company Overview

Since 1827, Norfolk Southern Corporation (NYSE: NSC) and its predecessor companies have safely moved the goods and materials that drive the U.S. economy. Today, it operates a customer‑centric and operations‑driven freight transportation network. Committed to furthering sustainability, Norfolk Southern helps its customers avoid 15 million tons of yearly carbon emissions by shipping via rail. Its dedicated team members deliver more than 7 million carloads annually, from agriculture to consumer goods, and is the largest rail shipper of auto products and metals in North America. Norfolk Southern also has the most extensive intermodal network in the eastern U.S., serving a majority of the country's population and manufacturing base, with connections to every major container port on the Atlantic coast as well as the Gulf of Mexico and Great Lakes. Learn more by visiting www.NorfolkSouthern.com.

At Norfolk Southern, we believe in celebrating our individuality. By leveraging the unique backgrounds and viewpoints of our employees, we can create a culture of innovation, respect, and inclusion. We know that employees thrive in a workplace where differing viewpoints, ideas, and experiences are freely shared and valued. As such, we encourage all employees to contribute their distinctive skills and capabilities to our organization.

Equal employment opportunities are available to all applicants regardless of race, color, religion, age, sex, national origin, disability status, genetic information, veteran status, sexual orientation, and gender identity. Together, we power progress.