Senior Data Engineer - Hybrid (Azure Databricks/Lakehouse)

LLP; Chicago (IL); Hybrid; USD 148,000 - 164,000; Full time

Benefits offered by this job

Bonus eligibility

Benefits program

Job summary

Sidley Austin LLP is seeking a Senior Data Engineer to design and maintain scalable data pipelines and lakehouse architectures. You will partner with analytics, product, and business teams to translate requirements into reliable data solutions, driving innovative data products.

Responsibilities include building end-to-end Azure Databricks data solutions, optimizing Delta Lake patterns, and mentoring junior engineers. Strong Python/SQL/Spark and governance experience are required.

Qualifications

• 5+ years of hands-on data engineering experience building scalable pipelines.
• Design, build, and operate data solutions on cloud data platforms (Azure/AWS/GCP).
• Strong Python, SQL, and Spark proficiency for data processing.
• Experience with metadata-driven ingestion frameworks using Python/Scala.
• Knowledge of CI/CD, monitoring, and data governance best practices.

Responsibilities

• Design, build, and maintain scalable data pipelines and infrastructure.
• Develop and optimize Delta Lake/Lakehouse patterns and schemas.
• Create high-quality PySpark/Spark SQL transformations and optimize performance.
• Implement data quality, validation, monitoring, and alerting mechanisms.
• Collaborate with data architects, analysts, scientists, and product teams.
• Leverage Azure/AWS/GCP cloud platforms for data storage and real-time processing.
• Mentor junior engineers and support governance and security practices.

Skills

Python

SQL

Apache Spark

Data modeling

CI/CD pipelines

Communication

Analytical thinking

Education

Bachelor's degree in Computer Science/Engineering/Data Science

Master's degree (preferred)

Tools

Azure Databricks

Delta Lake

ADLS Gen2

Azure Data Factory

Databricks Unity Catalog

Job description

Sidley Austin LLP is seeking a Senior Data Engineer to design and maintain scalable data pipelines and lakehouse architectures. You will partner with analytics, product, and business teams to translate requirements into reliable data solutions, driving innovative data products.

Responsibilities include building end-to-end Azure Databricks data solutions, optimizing Delta Lake patterns, and mentoring junior engineers. Strong Python/SQL/Spark and governance experience are required.