Senior Data Engineer - Hybrid (Azure & Databricks)

AbilityLinks; Birmingham (AL); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Southern Company Services is seeking a Data Engineer to build, operationalize, and maintain data pipelines across hybrid and multi-cloud environments. You will standardize data curation, taxonomy, and solution delivery for Databricks and Azure.

The role reports to the EOTS Leadership Team and supports multiple business areas. The ideal candidate has 9+ years in data engineering, strong Python and AI/ML experience, and expertise in on-prem and Azure tools.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Machine Learning, or related field
• 9+ years of experience in data engineering or related roles
• Experience with batch and real-time data processing frameworks
• Experience with big data technologies (e.g., Hadoop, HIVE, Spark)
• Experience with AI/ML solutions in Python
• 4+ years with on-prem tools (e.g., MSBI, Informatica, Oracle Golden Gate, SQL Server)
• 2+ years with Microsoft Azure tools (e.g., Data Lake, Data Factory, Databricks, Synapse, Power BI)
• Experience with containerization (e.g., Docker, OpenShift)

Responsibilities

• Design and develop methods to process structured, semi-structured, and unstructured data using batch and real-time techniques
• Develop release pipelines to automate build, version control, and deployment tasks
• Build and maintain reusable templates, code libraries, and semantic models
• Establish gatekeeping and guard rail processes for production data promotion
• Collaborate with cross‑functional teams including data, business, and technical stakeholders
• Promote data reusability and modular data product delivery
• Define and enforce data quality and integrity controls
• Provide technical guidance to less experienced data engineers
• Optimize performance using parallel processing and minimal data movement
• Stay current with data technologies and share insights across the organization
• Standardize, design, and develop Databricks applications, data integrations and automated business processes.
• Integrate data from multiple systems, databases, APIs, and business applications.
• Provide technical guidance, knowledge sharing, and support for business users and fellow data engineers.

Skills

Data pipelines

Databricks

SQL

Python

Azure

Spark

Hadoop

Data governance

Communication

Mentoring

Education

Bachelor's degree in Computer Science, Engineering, Machine Learning, or related field

Tools

Databricks

SQL Server

Informatica

Oracle GoldenGate

Power BI

Docker

OpenShift

MSBI

Data Factory

Synapse

Job description

Southern Company Services is seeking a Data Engineer to build, operationalize, and maintain data pipelines across hybrid and multi-cloud environments. You will standardize data curation, taxonomy, and solution delivery for Databricks and Azure.

The role reports to the EOTS Leadership Team and supports multiple business areas. The ideal candidate has 9+ years in data engineering, strong Python and AI/ML experience, and expertise in on-prem and Azure tools.