Python Developer

Spark Tek Management Consulting LLC; Chicago (IL); On-site; USD 100,000 - 250,000; Full time

Job summary

A leading consulting firm is seeking a Senior Databricks Developer to enhance their data engineering team in Chicago. The ideal candidate will have at least 8 years of experience in data engineering, with expertise in Databricks and big data technologies. This full-time, on-site role involves designing and maintaining data pipelines, ensuring data quality, and collaborating with various stakeholders to deliver high-performance data solutions.

Qualifications

• 8+ years of experience in data engineering or related roles.
• Extensive hands-on experience with Databricks, Apache Spark, and Delta Lake.
• Proficiency in Python, Scala, or SQL for data processing and scripting.

Responsibilities

• Design and implement scalable data pipelines using Apache Spark on Databricks.
• Develop solutions for large-scale data processing, transformation, and analytics.
• Collaborate with data architects, data scientists, and business stakeholders.

Skills

Databricks

Python

PySpark

SQL

Apache Spark

Delta Lake

Data lakehouse architecture

Cloud data platforms

Airflow

Azure Data Factory

DBT

Education

Bachelor’s or master's degree in computer science, Information Technology, Data Engineering, or a related field

Job description

Direct message the job poster from Spark Tek Management Consulting LLC

Delivery Manager | Talent Acquisition Expert | Connecting Top Talent with Exciting Opportunities | Building High-Performing Teams | Passionate About…

Note: This position is on W2 only. No C2C.

Job Role: Databricks and Python/PySpark Consultant

Location: Chicago, IL (Onsite – Local Candidates Preferred)

Employment Type: W2 only (Green Card holders)

Start Date: Immediate

Experience Required: Minimum 8+ years of genuine experience

Skills Required: Databricks, Python, PySpark, SQL

Position Overview:

We are seeking an experienced and highly motivated Senior Databricks Developer to join our growing data engineering team. The ideal candidate will have a minimum of 8 years of experience in data engineering with deep expertise in the Databricks Unified Analytics Platform, big data technologies, and cloud-based data architecture. This is a critical on-site role, responsible for designing, developing, and maintaining scalable and secure data pipelines and solutions.

Key Responsibilities:

• Design and implement scalable data pipelines using Apache Spark on Databricks.
• Develop solutions for large-scale data processing, transformation, and analytics.
• Collaborate with data architects, data scientists, and business stakeholders to deliver high-performance data solutions.
• Optimize Spark jobs and Databricks notebooks for performance and cost-efficiency.
• Leverage Delta Lake for ACID transactions and data versioning.
• Implement data quality frameworks and ensure adherence to data governance policies.
• Integrate Databricks with cloud platforms (AWS, Azure, GCP) and various data sources.
• Troubleshoot and resolve performance, connectivity, and data quality issues.

Qualifications:

• 8+ years of experience in data engineering or related roles.
• Extensive hands-on experience with Databricks, Apache Spark, and Delta Lake.
• Proficiency in Python, Scala, or SQL for data processing and scripting.
• Strong understanding of data lakehouse architecture and cloud data platforms (AWS, Azure, or GCP).
• Experience with orchestration tools such as Airflow, Azure Data Factory, or DBT.
• Solid background in data warehousing, data modeling, and ETL/ELT development.
• Familiarity with CI/CD and DevOps practices in a data environment is a plus.
• Excellent communication, collaboration, and problem-solving skills.

Education:

• Bachelor’s or master's degree in computer science, Information Technology, Data Engineering, or a related field.
• This is a full-time, on-site position. Candidates must be available to work from Chicago, IL

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Contract

Job function

• Industries

  Staffing and Recruiting

Referrals increase your chances of interviewing at Spark Tek Management Consulting LLC by 2x

Sign in to set job alerts for “Python Developer” roles.

Chicago, IL $150,000.00-$225,000.00 10 hours ago

Chicago, IL $150,000.00-$250,000.00 2 weeks ago

Chicago, IL $100,700.00-$180,000.00 2 weeks ago

Chicago, IL $130,000.00-$150,000.00 2 days ago

Software Engineer II - Full Stack Developer

Deerfield, IL $98,600.00-$157,500.00 1 month ago

Chicago, IL $120,000.00-$160,000.00 2 weeks ago

Python and Kubernetes Software Engineer - Data, AI/ML & Analytics

Python Software Engineer - Infrastructure

Chicago, IL $200,000.00-$250,000.00 1 day ago

Software Engineer (Python/Linux/Packaging)

Chicago, IL $80,000.00-$96,000.00 1 week ago

Chicago, IL $100,000.00-$155,000.00 3 months ago

Python and Kubernetes Software Engineer - Data, Workflows, AI/ML & Analytics

GoLang or Python Developer 3- Chicago, IL or Reston, VA- ONSITE (4 days a week)

Chicago, IL $145,000.00-$200,000.00 1 week ago

Chicago, IL $180,000.00-$190,000.00 2 weeks ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.