Databricks Data Engineer

VOLTO Consulting; Irving (TX); On-site; USD 120,000 - 150,000; Full time

Job summary

VOLTO Consulting is seeking a Databricks Data Engineer in Irving, TX, responsible for creating and maintaining data solutions in a Big Data environment. You will work with PySpark, design ETL pipelines, and ensure data quality.

The ideal candidate will have over 8 years of experience, strong skills in Databricks, and familiarity with cloud platforms like AWS and GCP. This role is onsite and offers opportunities to work with cutting-edge technologies.

Qualifications

• 8+ years of experience in data engineering.
• Strong knowledge of Databricks and PySpark.
• Experience with Big Data environments.

Responsibilities

• Design, develop, and maintain data solutions using Databricks.
• Collaborate with teams to gather requirements.
• Implement ETL processes for data migration.

Skills

PySpark

Python

Databricks Lakehouse

Databricks Certified Data Engineer Professional Certification

ML Ops

Big Data platforms

Cloud platforms (AWS, GCP)

System design patterns

Analytical skills

Problem-solving

Tools

Cloudera Hadoop

Docker

AWS EKS

Job description

Job Title

Databricks Data Engineer

Location

Irving, TX (Onsite)

Type

Full Time with TCS

Experience Required

8+ years

Job Description

As a Databricks Data Engineer, you will be responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing in a Big Data environment using predominantly PySpark/Python. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems using PySpark.

Roles & Responsibilities

• Collaborate closely with data scientists, data engineers, and business stakeholders to gather requirements and understand the business objectives driving data pipeline development.
• Design, develop, and maintain robust, scalable high-performance Data Pipelines using Databricks.
• Leverage Databricks features such as Lakehouse and Delta Lake for efficient data storage and Spark for distributed processing.
• Develop ETL/ELT pipelines using Databricks.
• Monitor pipeline health, troubleshoot data issues.
• Migrate on-prem PySpark, SAS data pipelines and ML Models to Databricks.
• Define and implement best practices in Databricks.
• Evaluate new Databricks features and tools, helping the organization stay at the forefront of innovation in AI and Big Data.
• Collaborate with cross-functional teams to identify and resolve data-related issues.

Qualifications

• Proven expertise in implementing Lakehouse and Delta Lake using Databricks.
• Strong PySpark and Python experience.
• Databricks Certified Data Engineer Professional Certification.
• Familiarity with ML Ops/LLM Ops and distributed systems.
• Experience with Big Data platforms like Cloudera Hadoop and cloud platforms like AWS, GCP.
• Solid understanding of system design patterns, scalability, observability, and performance tuning.
• Strong analytical and problem-solving skills.
• Passion for exploring and building with emerging technologies.

Good To Have Skills

• AWS EKS Experience, Docker and Containers.