W2 LEAD DATA ENGINEER(AWS, DATABRICKS)

Virisha LLC; Chicago (IL); Hybrid; USD 150,000 - 180,000; Full time

Job summary

Virisha LLC is seeking a Lead Data Engineer to design and implement an AWS Databricks Lakehouse platform, shaping analytics strategy and data governance. You will partner with IT, analytics, and business teams to deliver scalable solutions and drive decisions.

The role requires 10 years in IT with 4–6 years in data pipelines, Databricks on AWS, and strong SQL/Python skills. A leadership mindset and experience mentoring a team are a plus for this contract-to-hire position.

Qualifications

• 10 years of IT experience with deep data engineering & ETL.
• 4-6 years hands-on experience designing, building, and scaling data pipelines.
• Strong hands-on experience with Databricks (Spark, Delta Lake, Unity Catalog) and AWS (S3, IAM, networking).
• Proficiency in Databricks architecture combining data lakes and data warehouses.
• Familiar with Unity Catalog governance, DLT pipelines, Databricks SQL, and workflow orchestration.
• Expertise in SQL, Python, data modeling, and distributed processing.
• Experience with CI/CD, DevOps, and modern data architecture (Lakehouse, medallion).
• Agile analytics collaboration across stakeholders and strong communication.

Responsibilities

• Design and lead enterprise-grade pipelines for batch and streaming data processing.
• Own the full ETL/ELT lifecycle from design to production support.
• Drive data engineering standards, best practices, and architecture across the data ecosystem.
• Translate business requirements into scalable, high-performance data solutions.
• Mentor a small team while remaining hands-on.
• Ensure data reliability, governance, and cost optimization.

Skills

Databricks

AWS

SQL

Python

CI/CD

Data modeling

Delta Live Tables

Unity Catalog

Data lakehouse

ETL/ELT

Tools

Databricks

AWS

Unity Catalog

Delta Live Tables

Job description

Lead Data Engineer

Location : Chicagoland, IL candidates preferred, but work will be mostly remote with ability to come into Downtown office as needed.

Duration : 3-6 Month Contract to hire

Looking for 10 years of Data Engineering experience with the last 4 years working with Databricks on AWS, NOT ON AZURE OR ANOTHER PLATFORM.

Overview:

• We are investing in the next evolution of our data, analytics, and artificial intelligence ecosystem. We are looking for a Lead Data Engineer who's ready to do more than just build pipelines on Databricks and AWS.
• You will help shape enterprise analytics strategy, influence architecture, and partner directly with the business to drive smarter decisions.

Why This Role?

This is a high-impact, high visibility role where you will:

• Lead the design and implementation of a modern AWS Databricks Lakehouse platform
• Partner across IT, analytics, and business teams to turn data into decisions
• Drive scalable, governed, high-performance data solutions
• Influence how data powers analytics, reporting, and AI across the company

What You'll Be Doing

• Design and lead enterprise-grade pipelines, supporting both batch and streaming data processing
• Own the full ETL/ELT lifecycle, from design through production support
• Drive data engineering standards, best practices, and architecture across the data ecosystem
• Translate business requirements into scalable, high-performance data solutions
• Mentor a small team while remaining hands‑on
• Ensure data reliability, performance, governance, and cost optimization

Qualifications:

• 10 years of IT experience, including deep expertise in data engineering & ETL
• 4-6 years of recent hands‑on experience in designing, building, and scaling data pipelines
• Strong hands‑on experience with Databricks (Spark, Delta Lake, Unity Catalog) and AWS (S3, IAM, networking)
• Highly proficient in Databricks' core architectural pattern, which combines the best elements of data lakes and data warehouses.
• Familiar with advanced features like Unity Catalog (for governance), Delta Live Tables (DLT for pipelines), Databricks SQL, and workflow orchestration.
• Expertise in SQL, Python, data modelling, and distributed processing.
• Experience with CI/CD, DevOps, and modern data architecture (Lakehouse, medallion, etc.).
• Experience working in an Agile analytics environment where close collaboration with key stakeholders to address evolving requirements is the norm.
• Strong communication skills and the ability to translate complex technical concepts into solutions that deliver business outcomes.
• A passion for continued learning and an interest in staying current with the rapidly evolving data and analytics space.

What Sets You Apart

• You're analytical, curious, and solutions-oriented
• You thrive in environments where you balance priorities and drive outcomes
• You're comfortable with ambiguity and can quickly learn the business
• You build strong partnerships and bring a "get things done" mindset