Databricks Engineer

Capgemini; Cary (NC); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Healthcare coverage

Paid time off

Parental leave

401(k) plan

Employee Resource Groups

Job summary

Capgemini is seeking a Databricks Engineer with 8+ years of experience to design and optimize scalable data solutions on the Databricks Lakehouse Platform. You will implement Delta Lake, Medallion Architecture, and end-to-end data pipelines, collaborating with architects and analysts.

The role requires strong Python/SQL skills, cloud experience (Azure/AWS/GCP), and a track record in enterprise ETL/ELT. Onsite/Hybrid from Day 1 in Cary, NC with full benefits.

Qualifications

• Bachelor's degree in CS/IT/engineering or related field.
• 8+ years of IT experience with data engineering and big data.
• 5+ years hands-on Databricks experience.
• Strong Spark, PySpark, Spark SQL, and Delta Lake skills.
• Proficiency in Python and SQL.
• Experience in enterprise-scale ETL/ELT solutions.
• Understanding of data warehousing, modeling, and lakehouse architecture.
• Hands-on cloud experience (Azure, AWS, or GCP).
• Experience with Git and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines on Databricks.
• Build and optimize ETL/ELT workflows for batch and streaming data.
• Develop data transformation using PySpark and Spark SQL.
• Implement Delta Lake and Medallion Architecture for reliable data platforms.
• Integrate data from databases, APIs, cloud storage, and streams.
• Collaborate with data architects and analysts to deliver data-driven solutions.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Implement data quality, governance, security, and monitoring frameworks.
• Support CI/CD automation, release management, and production deployments.
• Troubleshoot production issues and provide platform support.
• Document technical designs and operational procedures.

Skills

Databricks

PySpark

Spark SQL

Python

SQL

ETL/ELT Development

Data Warehousing

Data Modeling

Git

CI/CD

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or related field

Tools

Databricks

Git

Terraform

Airflow

Kafka

Job description

Location : Cary, NC (Onsite/Hybrid from Day 1)

Experience : 8+ Years

Employee Type : Full Time with Benefits

Job Description

We are seeking a highly skilled Databricks Engineer with 8+ years of experience in Data Engineering, Big Data, and Cloud technologies. The ideal candidate will be responsible for designing, developing, and optimizing scalable data solutions on the Databricks Lakehouse Platform. This role requires deep expertise in Databricks, PySpark, Spark SQL, Delta Lake, and cloud-based data platforms to support enterprise analytics and data modernization initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT workflows for large-scale batch and streaming data processing.
• Develop data transformation and enrichment processes using PySpark and Spark SQL.
• Implement and manage Delta Lake and Medallion Architecture to support reliable and high-performance data platforms.
• Integrate data from multiple sources including databases, APIs, cloud storage, and streaming platforms.
• Collaborate with data architects, business analysts, and application teams to deliver data-driven solutions.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Implement data quality, governance, security, and monitoring frameworks.
• Support CI/CD automation, release management, and production deployments.
• Troubleshoot production issues and provide ongoing platform support.
• Document technical designs, processes, and operational procedures.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 8+ years of overall IT experience with strong expertise in Data Engineering and Big Data technologies.
• 5+ years of hands-on experience with Databricks.
• Strong experience with Apache Spark, PySpark, Spark SQL, and Delta Lake.
• Proficiency in Python and SQL programming.
• Experience designing and implementing enterprise-scale ETL/ELT solutions.
• Strong understanding of Data Warehousing, Data Modeling, and Lakehouse Architecture.
• Hands-on experience with cloud platforms such as Azure, AWS, or GCP.
• Experience with source control tools such as Git and CI/CD implementation.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and stakeholder management skills.

Required Technical Skills

Must Have

• Databricks
• PySpark
• Spark SQL
• Python
• SQL
• ETL/ELT Development
• Data Warehousing
• Data Modeling
• Git
• CI/CD

Preferred Skills

• Databricks Workflows
• Structured Streaming
• Kafka/Event Hubs
• AWS Glue
• Airflow
• Terraform
• Power BI or Tableau
• DevOps and Infrastructure as Code (IaC)

Life At Capgemini

Capgemini supports all aspects of your well-being throughout the changing stages of your life and career. For eligible employees, we offer:

• Healthcare including dental, vision, mental health, and well-being programs
• Financial well-being programs such as 401(k) and Employee Share Ownership Plan
• Paid time off and paid holidays
• Paid parental leave
• Family building benefits like adoption assistance, surrogacy, and cryopreservation
• Social well-being benefits like subsidized back-up child/elder care and tutoring
• Mentoring, coaching and learning programs
• Employee Resource Groups
• Disaster Relief

Disclaimer

Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity/expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by law.

This is a general description of the Duties, Responsibilities and Qualifications required for this position. Physical, mental, sensory or environmental demands may be referenced in an attempt to communicate the manner in which this position traditionally is performed. Whenever necessary to provide individuals with disabilities an equal employment opportunity, Capgemini will consider reasonable accommodations that might involve varying job requirements and/or changing the way this job is performed, provided that such accommodations do not pose an undue hardship.

Capgemini is committed to providing reasonable accommodations during our recruitment process. If you need assistance or accommodation, please reach out to your recruiting contact.