Cloud Data Platform Lead: Databricks, PySpark & Kafka

Envision Technology Solutions; Georgia; On-site; USD 140,000 - 180,000; Full time

Job summary

Envision Technology Solutions seeks an Experienced Data Technical Lead to design, develop, and support scalable cloud-based data platforms and streaming data pipelines. You will leverage Databricks, PySpark, Python, Java-based streaming, and cloud migration initiatives to deliver robust data solutions.

The role involves building Databricks ETL/ELT pipelines, real-time ingestion with Kafka, data movement across PostgreSQL and NoSQL stores, and reusable frameworks for ingestion and orchestration.

Qualifications

• Strong experience in Python and PySpark development.
• Hands-on experience with Azure Databricks and databricks SQL.
• Experience in Java-based streaming and ingestion frameworks.
• Strong knowledge of Apache Kafka streaming concepts.
• Experience working with PostgreSQL databases.
• Experience with YugabyteDB or Cassandra-based NoSQL databases.
• Strong SQL development and query optimization skills.
• Hands-on experience with GitLab CI/CD pipeline development and deployment automation.
• Understanding cloud-based data engineering and distributed processing concepts.
• Experience in data migration projects, especially Snowflake on-prem to Azure cloud migration.
• Experience designing enterprise-scale data lake or lakehouse architectures.
• Knowledge of streaming architectures and real-time analytics.
• Familiarity with cloud monitoring and observability tools.

Responsibilities

• Design and develop scalable Databricks ETL/ELT pipelines (Lakeflow & LakeBase) using Azure Databricks, PySpark, and Python.
• Implement real-time and batch data ingestion frameworks using Kafka and Java-based streaming solutions.
• Develop and optimize data processing workflows in Azure Databricks.
• Integrate and manage data movement between PostgreSQL, YugabyteDB, and cloud platforms.
• Build reusable frameworks for data ingestion, transformation, validation, and orchestration.
• Develop SQL-based data transformations, reporting datasets, and performance optimization solutions.
• Design and implement GitLab CI/CD pipelines for automated deployment, testing, and release management of Databricks notebooks, jobs, and data pipelines.
• Support Snowflake on-premises to Azure cloud migration initiatives.
• Ensure coding standards, performance tuning, monitoring, and operational stability of data pipelines.
• Develop Power BI dashboards and reports for business intelligence and analytics reporting.
• Develop API automation and integration solutions for data exchange between enterprise systems.

Skills

Python

PySpark

Databricks

Kafka

Java streaming

SQL

GitLab CI/CD

Azure Databricks

PostgreSQL

YugabyteDB

Snowflake migration

Cloud data engineering

Data lakehouse

Tools

GitLab

Power BI

Lakeflow

LakeBase

Databricks SQL

Snowflake

Job description

Envision Technology Solutions seeks an Experienced Data Technical Lead to design, develop, and support scalable cloud-based data platforms and streaming data pipelines. You will leverage Databricks, PySpark, Python, Java-based streaming, and cloud migration initiatives to deliver robust data solutions.

The role involves building Databricks ETL/ELT pipelines, real-time ingestion with Kafka, data movement across PostgreSQL and NoSQL stores, and reusable frameworks for ingestion and orchestration.