Hybrid Data Engineer Databricks, Azure, Real-Time Pipelines

Southern Company; Birmingham (AL); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Southern Company is seeking an experienced Data Engineer to build, operationalize, and maintain data pipelines across hybrid and multi-cloud environments. You will translate data into consumable formats and support batch and streaming workflows to enable analytics and AI use cases.

You will standardize data curation, taxonomy, and solution delivery for Databricks and Azure, and collaborate with cross‑functional teams on strategic data projects within Electric Operations Technology Solutions.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, ML, or related field.
• 9+ years of experience in data engineering or related roles.
• Experience with batch and real-time data processing frameworks.
• Experience with big data technologies (Hadoop, Hive, Spark).
• Experience with AI/ML solutions in Python.
• 4+ years with on-prem tools (MSBI, Informatica, Oracle GoldenGate, SQL Server).
• 2+ years with Microsoft Azure tools (Data Lake, Data Factory, Databricks, Synapse, Power BI).
• Experience with containerization (Docker, OpenShift).

Responsibilities

• Design and develop methods to process structured, semi-structured, and unstructured data using batch and real-time techniques.
• Develop release pipelines to automate build, version control, and deployment tasks.
• Build and maintain reusable templates, code libraries, and semantic models.
• Establish gatekeeping and guard rail processes for production data promotion.
• Collaborate with cross-functional teams including data, business, and technical stakeholders.
• Promote data reusability and modular data product delivery.
• Define and enforce data quality and integrity controls.
• Provide technical guidance to less experienced data engineers.
• Optimize performance using parallel processing and minimal data movement.
• Stay current with data technologies and share insights across the organization.
• Standardize, design, and develop Databricks applications, data integrations and automated business processes.
• Integrate data from multiple systems, databases, APIs, and business applications.
• Provide technical guidance, knowledge sharing, and support for business users and fellow data engineers.

Skills

Data Modeling

Databricks

Python

SQL

Azure

Big Data

CI/CD

Data Governance

Education

Bachelor's degree in CS/Engineering/related

Tools

Hadoop

Spark

Informatica

Databricks

SQL Server

Docker

OpenShift

Job description

Southern Company is seeking an experienced Data Engineer to build, operationalize, and maintain data pipelines across hybrid and multi-cloud environments. You will translate data into consumable formats and support batch and streaming workflows to enable analytics and AI use cases.

You will standardize data curation, taxonomy, and solution delivery for Databricks and Azure, and collaborate with cross‑functional teams on strategic data projects within Electric Operations Technology Solutions.