Lead Data Engineer

US staffing Inc; Chicago (IL); On-site; USD 120,000 - 180,000; Full time

Job summary

US staffing Inc. is seeking a senior data engineer in Chicago to design, implement, and optimize large-scale data pipelines. The role emphasizes Databricks-based lakehouse architectures and AWS cloud services, including S3 and IAM, with a focus on scalable, cost-efficient ETL processes.

The candidate will collaborate with analytics stakeholders, lead data modeling efforts, and drive best practices across CI/CD and DevOps in an Agile analytics environment.

Qualifications

• 10+ years of IT experience with data engineering and ETL.

Responsibilities

Skills

Databricks

AWS

SQL

Python

Data modeling

Distributed processing

CI/CD

DevOps

Agile analytics

Stakeholder collaboration

Tools

Databricks

Unity Catalog

Delta Live Tables

Databricks SQL

Job description

Work Authorization: USC & GC

• 10+ years of IT experience, including deep expertise in data engineering & ETL.
• 4-6+ years of recent hands-on experience in designing, building, and scaling data pipelines.
• Direct experience designing, configuring, coding, and deploying systems.
• Strong, hands-on experience with Databricks (Spark, Delta Lake, Unity Catalog) and AWS (S3, IAM, networking).
• Highly proficient in Databricks' core architectural pattern, which combines the best elements of data lakes and data warehouses.
• Cost and performance management: Databricks can become highly expensive if left unoptimized. This person must know how to choose the right AWS EC2 instance types for clusters, configure auto-scaling efficiently, use Spot instances safely, minimize data shuffling, and optimize Photon engine performance to get the fastest runtimes for the lowest cost.
• Familiar with advanced features like Unity Catalog (for governance), Delta Live Tables (DLT for pipelines), Databricks SQL, and workflow orchestration.
• Expertise in SQL, Python, data modeling, and distributed processing.
• Experience with CI/CD, DevOps, and modern data architecture (lakehouse, medallion, etc.).
• Experience working in an Agile analytics environment where close collaboration with key stakeholders to address evolving requirements is the norm.
• Excellent communicator who can be the front-face of the data team and lead and push back on stakeholders during requirements gathering, delivery, and support.
• A passion for continued learning and an interest in staying current with the rapidly evolving data and analytics space.