Senior Data Engineer (Databricks)

Gulf Coast Automation Group

Chicago (IL)

Hybrid

USD 150,000 - 164,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
401(k)
Paid time off

Job summary

TalentFish is seeking a Senior Data Engineer to build a brand-new Azure Databricks platform in Chicago, IL. You will own end-to-end data pipelines, design scalable models, and enforce governance standards on a hybrid schedule with on-site and remote work options.

You will write Python, SQL, and Spark code daily, mentor peers, and collaborate across data, BI, and finance teams to deliver fast, reliable data for analytics and AI use cases.

Qualifications

  • Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.
  • 6+ years of hands-on data engineering experience with end-to-end pipelines (ingestion, validation, transformation, governance).
  • Databricks experience (Azure preferred; AWS/GCP acceptable) including Delta Lake, Unity Catalog, Databricks SQL, Spark clusters, Workflows/Jobs.
  • Strong coding skills in Python, SQL, and Apache Spark.
  • Proven experience designing and building scalable ETL/ELT pipelines; reusable, metadata-driven pipelines.
  • Experience with object storage, lakehouse engines, orchestration tools, and streaming or CDC pipelines.
  • Solid data modeling, schema design, and performance tuning.
  • Git-based workflows, CI/CD, test automation, monitoring and alerting.
  • Troubleshooting skills; focus on scalability, reliability, and maintainability.
  • Ability to clearly explain what you built and its business impact.

Responsibilities

  • Lead the design and build of scalable data pipelines and models in Azure Databricks on a brand-new platform with no production code yet.
  • Own pipelines end to end from ingestion to transformation and outputs for analytics, BI, and AI/ML use cases.
  • Refactor and migrate legacy data warehouses into the centralized Databricks lakehouse.
  • Work daily in Python, SQL, and Spark with Databricks assets, workflows, and API-driven jobs.
  • Establish engineering standards for CI/CD (GitHub/Azure DevOps), automated data testing, monitoring and alerting.
  • Shape and enforce standards for data modeling and governance.
  • Troubleshoot and optimize performance across the platform.
  • Mentor junior engineers, conduct code reviews, and lead by example in design and testing practices.
  • Partner with Data Engineering, BI, AI, Finance, and product owners to translate technical work into business impact.

Skills

Databricks (Azure)
Python
SQL
Apache Spark
Delta Lake
Unity Catalog
ETL/ELT pipelines
GitHub / Azure DevOps CI/CD
Data modeling
CI/CD tooling

Education

Bachelor's degree in Computer Science, Engineering, Data Science, or related field

Tools

Azure Databricks
GitHub

Job description

Job Title: Senior Data Engineer (Databricks)
Primary Location: Chicago, IL (Hybrid – 3 days onsite / 2 days remote)
Position Type: Full-Time, Direct Hire

OVERVIEW

TalentFish is casting a line for a Senior Data Engineer. This is a full-time, direct hire role in Chicago, IL, working a hybrid schedule with a minimum of three days onsite per week. This role exists as part of a multi-year initiative to transform our client's Data & AI function, building a modern data platform that enables speed-to-insight and makes the organization truly data-driven.
You'll be one of the core builders of a brand-new Azure Databricks platform, designing and delivering the pipelines, transformations, and models that everything else runs on. The environment is greenfield: Databricks is stood up, but no production pipelines exist yet, so you'll help set the standards from day one. This is a hands‑on senior individual contributor role, not a people management, architecture, AI/LLM, or BI/reporting role. The architecture is defined; your job is to build it well and raise the technical bar of a growing team (expanding to 25–30) that is largely BI-focused today.
The work carries strong executive sponsorship and enterprise-wide visibility, with the chance to influence engineering practices across the organization. The interview process includes a live technical assessment covering Python, SQL, and practical data pipeline logic.

WHAT YOU BRING TO THE ROLE. (IDEAL EXPERIENCE)
  • Bachelor's degree in Computer Science, Engineering, Data Science, or a related field
  • 6+ years of hands‑on data engineering experience, with a track record of owning pipelines end to end: ingestion, validation, transformation, and governance
  • Real, hands‑on Databricks experience (Azure preferred; AWS or GCP acceptable), including Delta Lake, Unity Catalog, Databricks SQL, Spark clusters, and Workflows/Jobs
  • Strong coding skills in Python, SQL, and Apache Spark that go well beyond basic scripting
  • Proven experience designing and building scalable ETL/ELT pipelines, including reusable, metadata‑driven pipelines
  • Experience with object storage, lakehouse engines, orchestration tools, and streaming or CDC pipelines
  • Solid data modeling, schema design, and performance tuning experience
  • Working knowledge of Git‑based workflows, CI/CD, test automation, and monitoring and alerting
  • Strong troubleshooting skills and a solid grasp of engineering fundamentals: scalability, reliability, and maintainability
  • The ability to clearly explain what you built, why it mattered, and how it solved a business or platform problem
Preferred
  • Experience building or migrating a data platform from the ground up
  • Infrastructure as Code (Terraform, Bicep, ARM) and Governance as Code experience
  • Multi‑cloud background with depth in Azure
  • Legacy data warehousing and/or Hadoop experience
  • Deep Databricks exposure beyond basic pipelines, such as enterprise‑scale work and optimization
  • A track record of mentoring and upskilling peers
  • Agile delivery experience
WHAT YOU'LL DO. (SKILLS USED IN THIS POSITION)
  • Lead the design and build of scalable data pipelines and models in Azure Databricks on a brand-new platform with no production code yet
  • Own pipelines end to end, from ingesting source systems to transforming and modeling data to producing outputs for downstream analytics, BI, and AI/ML use cases
  • Refactor and migrate multiple legacy data warehouses into the centralized Databricks lakehouse
  • Work at the code level every day in Python, SQL, and Spark, building pipelines with Databricks Asset Bundles, Spark workflows, and API‑driven jobs
  • Establish engineering standards for CI/CD (GitHub/Azure DevOps), automated data testing, monitoring, and alerting
  • Help shape and enforce standards for data modeling and governance
  • Troubleshoot and optimize performance across the platform
  • Mentor and coach junior engineers, conduct code reviews, and lead by example in design, coding, and testing practices
  • Partner with Data Engineering, BI, AI, Finance, and product owners, and help translate technical work into business impact
COMPENSATION INFORMATION

The expected salary range for this position i s $150,000-$164,000 per year, depending on experience and qualifications. This role also qualifies for comprehensive benefits such as health insurance, 401(k), and paid time off. TalentFish is committed to pay transparency and equal opportunity. The salary range provided is in compliance with applicable state and federal regulations.
This role requires authorization to work in the U.S. without current or future visa sponsorship.
All offers are contingent upon the completion of a background check, which may include but is not limited to reference checks, education verification, employment verification, drug testing, criminal records checks, and any required certifications or compliance requirements based on the end client's background check policies and applicable laws.
TalentFish is an employee‑owned company pioneering a new realm in talent acquisition. We are redefining IT staffing by evolving AI, video screening, and our unique platform. TalentFish focuses on providing the best employee, consultant, and client experience possible. At TalentFish we are an Equal Opportunity Employer; we embrace and encourage diversity!

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Gulf Coast Automation Group • Chicago (IL)

Hybrid
USD 150,000 - 164,000
Health insurance
401(k)
Paid time off
Senior Data Engineer
Senior Data Engineer

Burtch Works • Chicago (IL)

Hybrid
USD 148,000 - 164,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

CGI Group, Inc. • Miami (FL)

Hybrid
USD 92,000 - 204,000
Competitive compensation
Comprehensive insurance options
401(k) match and share purchase plan
+3
Senior Data Architect and Engineer
Senior Data Architect and Engineer

Gulf Coast Automation Group • Chicago (IL)

Hybrid
USD 165,000 - 220,000
Specialist Solutions Architect - Data Engineering & Warehousing (Financial Services)
Specialist Solutions Architect - Data Engineering & Warehousing (Financial Services)

WinsAbove • United States

Hybrid
USD 149,000 - 248,000
Benefits
Solutions Architect - Digital Native Business (Geo)
Solutions Architect - Digital Native Business (Geo)

Databricks Inc. • United States

Remote
USD 180,000 - 248,000
Senior Data & AI Solutions Architect
Senior Data & AI Solutions Architect

Neon • California

On-site
USD 182,000 - 250,000
Remote: Data Engineering & Lakehouse Solutions Architect
Remote: Data Engineering & Lakehouse Solutions Architect

WinsAbove • United States

Hybrid
USD 149,000 - 248,000
Benefits
Sr. Solutions Engineer - Digital Native Business (Strategic Accounts)
Sr. Solutions Engineer - Digital Native Business (Strategic Accounts)

Databricks Inc. • Oregon

On-site
USD 152,000 - 209,000
Sr. Solutions Engineer - Digital Native Business (Strategic Accounts)
Sr. Solutions Engineer - Digital Native Business (Strategic Accounts)

Databricks Inc. • Washington

On-site
USD 152,000 - 209,000