Senior Data Engineer

TechDigital Group; Chicago (IL); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading technology company is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines and solutions using Databricks, Python, and DevOps practices. Candidates should have extensive experience with cloud-based data platforms and data engineering. This role will involve architecting data pipelines, developing ETL workflows, and automating processes, alongside ensuring data governance and compliance. Strong problem-solving skills are essential.

Qualifications

• 8+ years in data engineering or related roles.
• Expert-level Python programming for data processing and automation.
• Hands-on experience with Databricks and Delta Lake.
• Strong knowledge of Azure Data Factory, Data Lake, and Synapse (or equivalent cloud platforms).
• Proficiency in SQL and performance tuning.
• Familiarity with DevOps practices, CI/CD, and infrastructure-as-code tools.
• Experience with data migration and cloud-native architectures.
• Excellent problem-solving and communication skills.

Responsibilities

• Architect and implement data pipelines using Databricks and Delta Lake for batch and streaming data.
• Develop and maintain ETL/ELT workflows integrating diverse data sources into cloud data lakes and warehouses.
• Implement data governance, security, and compliance standards.
• Automate workflows using Python scripting and CI/CD pipelines.
• Troubleshoot and resolve issues in data ingestion, transformation, and storage.

Job description

The Senior Data Engineer will design, build, and optimize scalable data pipelines and solutions leveraging Databricks, Python, ETL, DevOps. This role requires strong experience in cloud-based data platforms and advanced data engineering practices to support analytics and business intelligence initiatives.

Key Responsibilities

• Architect and implement data pipelines using Databricks and Delta Lake for batch and streaming data.
• Develop and maintain ETL/ELT workflows integrating diverse data sources into cloud data lakes and warehouses.
• Implement data governance, security, and compliance standards.
• Automate workflows using Python scripting and CI/CD pipelines.
• Troubleshoot and resolve issues in data ingestion, transformation, and storage.

Required Skills & Experience

• 8+ years in data engineering or related roles.
• Expert-level Python programming for data processing and automation.
• Hands‑on experience with Databricks and Delta Lake.
• Strong knowledge of Azure Data Factory, Data Lake, and Synapse (or equivalent cloud platforms).
• Proficiency in SQL and performance tuning.
• Familiarity with DevOps practices, CI/CD, and infrastructure-as-code tools.
• Experience with data migration and cloud-native architectures.
• Excellent problem-solving and communication skills