Senior Data Engineer

Burtch Works; Chicago (IL); Hybrid; USD 148,000 - 164,000; Full time

Job summary

Burtch Works seeks a hands-on Senior Data Engineer in Chicago to design, build and maintain data pipelines and infrastructure powering analytics and ML across the firm. You will own pipelines end-to-end from ingestion to output in Databricks, writing code daily in Python, SQL and Spark.

This role requires 6+ years of data engineering experience, strong Databricks and cloud skills, and collaboration with stakeholders while upholding CI/CD and data governance practices.

Qualifications

• Bachelor's degree in CS/Engineering/Data Science or related field.
• 6+ years of hands-on data engineering experience.
• Proficiency in Python, SQL and Spark; live code interview.
• Experience with Azure Databricks and cloud data platforms.

Responsibilities

• Build end-to-end data solutions on Azure Databricks.
• Design, develop and maintain scalable ETL/ELT and streaming pipelines.
• Develop PySpark and Spark SQL transformations; optimize performance.
• Refactor and migrate data warehouses to the Databricks platform.
• Own pipeline orchestration, monitoring and CI/CD with testing and security checks.
• Implement data quality frameworks and monitoring.
• Troubleshoot Databricks platform and pipeline issues.
• Collaborate with cross-functional teams to align data work with business goals.
• Mentor junior engineers and promote best practices.

Skills

Python

SQL

Apache Spark

Databricks

Azure

ETL/ELT

Data Modeling

CI/CD

Git

PySpark

Education

Bachelor's degree in Computer Science, Engineering, Data Science or related field

Master's degree in Computer Science or Engineering

Tools

Azure Databricks

Delta Lake

ADLS Gen2

Terraform

Unity Catalog

MLflow

Notebooks

Job description

Job Title: Senior Data Engineer

Location: Chicago, IL (hybrid, minimum three days onsite). Candidates must reside in the Chicagoland area.

Employment Type: Permanent, full-time

Target Compensation: $148,000 to $164,000

About The Company

Our client is a leading global law firm. Its Data & AI function is in a multi-year transformation to build modern data platforms, speed up access to insight and make the firm data-driven. The Data Engineering team is growing to 25–30 people and is building a new Azure Databricks platform from the ground up, including migrating existing data warehouses into it.

Job Summary

We are seeking a hands-on Senior Data Engineer to design, build and maintain the data pipelines, models and infrastructure that power analytics, business intelligence and machine learning across the firm. This is a senior individual contributor role. You will own pipelines end to end, from ingestion through transformation and modeling in Databricks to output for downstream use, and help set the team's engineering standards for CI/CD, testing, monitoring and alerting. The role reports to the Senior Manager, Data Engineering and works closely with the Director of Data Platform & Engineering.

This is not a people management, solution architect, AI/LLM, data science or BI-only role. The architecture is defined; this role builds within it and writes code daily in Python, SQL and Spark.

Key Responsibilities

• Build end-to-end data solutions on Azure Databricks.
• Design, develop and maintain scalable ETL/ELT and streaming pipelines using Apache Spark, Delta Lake and Azure Data Lake Storage (ADLS Gen2).
• Build and optimize Delta Lake / Lakehouse patterns (Bronze/Silver/Gold), including schema evolution and time travel.
• Develop high-quality PySpark and Spark SQL transformations, and optimize joins, partitioning, caching and shuffle behavior.
• Build and optimize data models and schemas for analytics, reporting and operational data stores.
• Refactor and migrate existing data warehouses into the centralized Databricks platform.
• Own pipeline orchestration, monitoring, testing and CI/CD, supported by version control, linting, automated testing, security scanning and monitoring.
• Implement and maintain data quality frameworks, including validation, monitoring and alerting.
• Troubleshoot and resolve complex Databricks platform, infrastructure and pipeline issues to minimize downtime and maintain performance.
• Contribute to data governance practices that support data integrity, accessibility and compliance with regulations such as GDPR.
• Collaborate with data architects, analysts, data scientists, BI, finance, product owners and business stakeholders to align data engineering work with business goals.
• Mentor and coach junior engineers, conduct code reviews and promote best practices in software engineering, data engineering and agile development.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science or a related field.
• 6+ years of hands-on data engineering experience designing, building and operating scalable data pipelines and ETL/ELT processes.
• Hands-on Databricks experience building production pipelines (Azure preferred; AWS or GCP acceptable).
• Strong proficiency in Python, SQL and Apache Spark. Skills are assessed in a live technical interview.
• Experience owning pipelines end to end (ingestion, validation, transformation, governance) and clearly explaining the architecture behind them.
• Experience building reusable, metadata-driven data ingestion frameworks using Python and Scala.
• Extensive experience with cloud data platforms in Azure, AWS or Google Cloud.
• Hands-on experience with modern data platform components: object storage, lakehouse engines, orchestration tools, columnar warehouses and streaming services.
• Experience building streaming or CDC pipelines.
• Proven experience with data modeling, schema design and performance tuning of large-scale data systems.
• Deep understanding of data engineering best practices: Git-based code repositories, CI/CD pipelines, test automation, monitoring and alerting.
• Experience presenting data through tables, reports, dashboards and other visualization tools.
• Strong problem-solving and analytical skills with close attention to detail.
• Excellent communication skills and experience working with technical and business stakeholders.

Preferred Qualifications

• Master's degree in Computer Science or Engineering.
• Experience building data pipelines in an Azure Databricks environment.
• Knowledge of Databricks architecture and core components, including Lakehouse, Delta Lake, Databricks SQL, Spark clusters, Unity Catalog, Databricks Workflows (Jobs), Asset Bundles, MLflow and Notebooks.
• Experience integrating Azure Databricks with Azure DevOps, Azure Blob Storage / ADLS Gen2, Azure Key Vault and Azure Data Factory.
• Familiarity with enterprise data modeling tools such as ERwin Data Modeler, including applying logical and physical data models to analytical and lakehouse architectures.
• Experience building data platforms from the ground up or migrating to them.
• Experience with Infrastructure as Code (Terraform, Bicep or ARM) and Governance as Code.
• Multi-cloud background with depth in Azure.
• Legacy data warehousing or Hadoop experience.
• Familiarity with machine learning workloads and partnering on feature engineering.
• Experience mentoring and upskilling peers.
• Experience working in an Agile delivery model.

Other Skills And Abilities

• Strong organizational skills and attention to detail.
• Good judgment and strong analytical and problem-solving skills.
• Strong interpersonal communication skills and the ability to work well with others.
• Able to preserve confidentiality and exercise discretion.
• Able to work under pressure and manage multiple projects with competing deadlines and priorities.

Work Arrangement

Hybrid, with a minimum of three days per week in the Chicago office. Regular, reliable attendance is required. Additional hours may be required during periods of heavy workload, and the daily schedule needs flexibility to meet firm requirements.

Interview Process

Hiring manager interview (1 hour, virtual), live technical assessment covering Python, SQL and data pipeline logic (1 hour, virtual), and a final onsite interview (1 hour).

Applicants must be authorized to work in the United States without the need for employer sponsorship, now or in the future.