Databricks + PySpark Engineer | Systegration | Waukegan, IL

Systegration; Waukegan (IL); Vor Ort; USD 120.000 - 180.000; Vollzeit

Zusammenfassung

Systegration is seeking an experienced Databricks + PySpark Engineer for a 100% on-site role in Waukegan, IL. You will design, build, and optimize large-scale data pipelines, lakehouse architectures, and enterprise ETL frameworks using Databricks, PySpark, and advanced SQL.

You will collaborate with data architects and BI teams, perform performance tuning, ensure data governance, and contribute to cost governance across distributed clusters while maintaining strict on-site requirements.

Qualifikationen

• Bachelor's degree in CS/IT or related field.
• 5+ years in big data engineering, data warehousing, and ETL development.
• Strong Databricks and PySpark expertise.
• On-site work in Waukegan, IL; no OPTs/CPTs.

Aufgaben

• Design, implement, test, and deploy scalable data pipelines with Databricks and PySpark.
• Optimize lakehouse architectures and ETL workflows for enterprise environments.
• Collaborate with data architects and BI teams to gather requirements.
• Perform performance tuning and cost governance across clusters.
• Maintain data governance, security baselines, and compliance.
• Document pipelines and runbooks; participate in design reviews.

Kenntnisse

Databricks

PySpark

Advanced SQL

ETL pipelines

Data warehousing

Lakehouse architectures

Performance tuning

Ausbildung

Bachelor's degree in Computer Science, Data Science, IT, or related

Tools

Informatica

Git

CI/CD tooling

Terraform

Spark

Jobbeschreibung

Position Summary

Systegration is actively seeking an experienced, high-caliber, and technical Databricks + PySpark Engineer with a strong professional background in big data analytics, distributed data pipelines, and enterprise ETL architectures to join our direct client project on a full-time contract-to-hire (C2H) basis 100% on-site in Waukegan, Illinois (local Waukegan area IL or WI candidates only). In this critical data engineering role, you will be responsible for designing, building, optimizing, and maintaining high-performance data pipelines, lakehouse architectures, and scalable analytics solutions. You will leverage your deep specialist expertise in Databricks, PySpark, advanced SQL, and enterprise ETL frameworks to deliver uncompromised data reliability, high performance, and operational excellence. This opportunity offers competitive compensation, executive technical visibility, and exceptional career growth prospects within a premier global IT staffing and digital consulting enterprise.

Detailed Job Description

As a Databricks + PySpark Engineer supporting Systegration in Waukegan, Illinois, you will take full operational, tactical, and technical ownership of distributed data pipeline development, Databricks lakehouse optimization, ETL execution, and data warehouse integration across mission-critical enterprise environments. Your day-to-day responsibilities include designing, developing, testing, and deploying scalable data processing pipelines utilizing Databricks and PySpark, writing highly optimized Spark SQL queries and complex data transformation logic for large-scale datasets, collaborating closely with client stakeholders, data architects, and BI teams to gather requirements and deliver robust analytics solutions, conducting rigorous performance tuning, query optimization, and cost governance across distributed cluster environments, managing data migration and integration tasks from legacy systems (with Informatica experience serving as a strong professional advantage), participating in client interviews (including online screening and potential on-site assessments), ensuring strict adherence to data governance, security baselines, and compliance standards, and managing application requirements by sending your updated resume via direct email to Ben Golub at [e-mail]. You will utilize your proactive troubleshooting and analytical problem-solving skills to resolve complex data engineering anomalies under strict project delivery timelines. Furthermore, you will manage application requirements ensuring strict compliance with the 100% on-site requirement in Waukegan (IL/WI local candidates only) and eligible work authorization (no OPTs/CPTs).

Key Responsibilities

• Design, architect, develop, test, and maintain scalable big data pipelines and lakehouse architectures utilizing Databricks.
• Write high-performance, robust PySpark code and complex Spark SQL queries for large-scale distributed data processing.
• Build, monitor, and optimize enterprise ETL workflows, data ingestion mechanisms, and data transformation frameworks.
• Conduct rigorous performance tuning, cluster optimization, and query cost reduction across cloud data platforms.
• Collaborate seamlessly with cross-functional data architects, BI analysts, and client stakeholder squads in Agile sprints.
• Manage integration tasks and data migration workflows from legacy data warehouses (including Informatica environments).
• Maintain comprehensive technical documentation of data pipelines, architectural schemas, and operational runbooks.
• Participate actively in technical design reviews, code profiling, and multi-stage interview workflows.

Required Qualifications & Skills

• Bachelor's degree in Computer Science, Data Science, Information Technology, Software Engineering, or a related discipline.
• Minimum of 5+ years of progressive professional experience in big data engineering, data warehousing, and ETL development.
• Strong, demonstrable hands‑on professional expertise and subject matter expertise in Databricks platform architecture.
• Comprehensive working knowledge and practical proficiency in PySpark development for distributed data processing.
• Deep practical proficiency in advanced SQL querying, performance tuning, and relational database management systems.
• Solid practical experience in building, scaling, and maintaining enterprise ETL pipelines and data lakehouse solutions.
• Exceptional analytical, troubleshooting, debugging, and problem‑solving capabilities under high‑pressure conditions.
• Excellent written and verbal English communication skills with strong stakeholder coordination and collaboration abilities.
• Mandatory legal work eligibility (no OPTs/CPTs) and physical availability for 100% on‑site operations in Waukegan, IL (local IL or WI candidates only).

Nice-to-Have Skills

• Professional certifications such as Databricks Certified Professional Data Engineer or Cloud Data Platform certifications (AWS/Azure/GCP).
• Prior professional working experience in top‑tier financial institutions, healthcare enterprises, or global IT consulting firms.
• Hands‑on familiarity with legacy ETL tools, specifically Informatica Power Center or Informatica Cloud (IICS).
• Experience utilizing version control systems (Git), CI/CD deployment pipelines, and infrastructure‑as‑code tooling (Terraform).
• Exposure to streaming data architectures (Kafka, Spark Streaming) and delta lake optimization techniques.

Application Information

• Salary/Rate: Open / Based on Qualifications (Competitive C2H Package)
• Deadline: Open until filled (Urgent Waukegan On‑Site Contract‑to‑Hire Opening)
• Notice Period: Immediate / Available to Start ASAP
• Contract Duration: Contract‑to‑Hire (12 Months, Possibly Longer) / 100% On‑Site (Waukegan, IL)
• Company Website: https://www.systegration.com