Senior Azure Databricks Data Engineer

Valiant Technologies LLC; Houston (TX); On-site; USD 110,000 - 160,000; Full time

Job summary

VALIANT TECHNOLOGIES LLC is seeking a Specialist with 7 years of experience to design and develop scalable data solutions using Azure Databricks. You will design, develop, and optimize scalable data pipelines to support enterprise-level data processing and analytics.

You will write PySpark and Python code to support data ingestion, transformation and analytics, troubleshoot pipelines, and collaborate with data architects, analysts and stakeholders to align solutions with business objectives.

Qualifications

• 7 years of experience designing and developing scalable data solutions.
• Proficient in PySpark and Python for data ingestion, transformation and analytics.
• Experience building and optimizing Azure Databricks pipelines and workflows.
• Ability to collaborate with data architects, analysts and stakeholders.

Responsibilities

• Lead the design, development and enhancement of complex datapipelines using AzureDatabricks.
• Write clean, maintainable PySpark and Python code for data ingestion, transformation and analytics.
• Troubleshoot datapipelines and workflows to ensure minimal downtime and high availability.
• Collaborate with data architects, analysts and stakeholders to gather requirements and deliver scalable data solutions.
• Mentor and guide junior team members on AzureDatabricks and best practices in data engineering.
• Document data processes, architectures and workflows to ensure knowledge sharing and compliance.

Skills

Azure Databricks Experience

PySpark & Python

Data Engineering

Tools

Azure Databricks

PySpark

Python

ETL

Job description

VALIANT TECHNOLOGIES LLC is seeking a Specialist with 7 years of experience to design and develop scalable data solutions using Azure Databricks. You will design, develop, and optimize scalable data pipelines to support enterprise-level data processing and analytics.

You will write PySpark and Python code to support data ingestion, transformation and analytics, troubleshoot pipelines, and collaborate with data architects, analysts and stakeholders to align solutions with business objectives.