Senior Data Engineer - Databricks Platform Lead (Hybrid)

Fairweather, LLC; Chicago (IL); Hybrid; USD 120,000 - 160,000; Full time

Benefits offered by this job

Health/Vision/Dental/Life Insurance

401k with company match

Flexible time off

Job summary

iManage is seeking a Senior Data Engineer in Chicago, IL to own the iMData platform and pipelines, collaborating with ML engineers to deliver scalable data insights. You will mentor junior engineers, engage with stakeholders, and drive data-driven decisions.

The role emphasizes building on Databricks, Delta Lake, and medallion architectures, with CI/CD and Terraform for infrastructure. A flexible, hybrid environment balances office collaboration with remote work.

Qualifications

• A Bachelor’s degree in Computer Science or equivalent field.
• 5-7 years of experience in data engineering with production-grade pipelines.
• Hands-on production experience with Databricks, Delta Lake, and medallion architecture.
• Familiarity with Terraform for infrastructure as code.
• Understanding of CI/CD practices, specifically using GitHub Actions and Databricks Asset Bundles (DABs).
• Deep expertise in PySpark, tuning and scaling distributed workloads.
• Strong data modeling skills, including star schemas in gold layers.
• Independent problem solver and strong collaborator across teams.

Responsibilities

• Using PySpark to extract, transform, and load data into our Enterprise Data Platform using a medallion architecture.
• Designing gold layer star schemas to support AI, analytical, and reporting workloads.
• Serving as a technical lead and mentor for junior and mid-level data engineers on the team.
• Defining and advocating for engineering best practices: code standards, testing, documentation, and pipeline design patterns.
• Learning and using system‑specific APIs to extract data from enterprise SaaS applications, including Salesforce and Zendesk.
• One time migration of pipelines, Spark notebooks, triggers, data models, and Power BI dashboards from Microsoft Synapse to Azure Databricks.
• Gathering data requirements from various business stakeholders and translating into deliverables.
• Working in an agile fashion aligned with our team Agile Methodologies.
• Applying best practices to ensure the security and privacy of the data repositories.
• Developing CI/CD workflows to enhance our SDLC.
• Maintaining and improving our Infrastructure as Code using Terraform.

Skills

PySpark

Databricks

Delta Lake

Terraform

CI/CD

GitHub Actions

Databricks Asset Bundles

Data modeling

Mentoring

Agile

Education

Bachelor's degree in Computer Science or equivalent

Tools

Terraform

Databricks Asset Bundles (DABs)

GitHub Actions

Job description

iManage is seeking a Senior Data Engineer in Chicago, IL to own the iMData platform and pipelines, collaborating with ML engineers to deliver scalable data insights. You will mentor junior engineers, engage with stakeholders, and drive data-driven decisions.

The role emphasizes building on Databricks, Delta Lake, and medallion architectures, with CI/CD and Terraform for infrastructure. A flexible, hybrid environment balances office collaboration with remote work.