Senior PySpark & Databricks Data Engineer — Hybrid

InfoSmart Technologies, Inc; Atlanta (GA); Hybrid; USD 120,000 - 150,000; Full time

Job summary

InfoSmart Technologies, Inc. is seeking a Data Engineer to design, build, and maintain scalable data pipelines using PySpark and Databricks, with a focus on streaming data via Kafka and orchestrating workflows with Airflow.

The role requires hands-on AWS data warehousing on Redshift, building cloud-native solutions with S3, Glue, Lambda, EMR, IAM, and CloudWatch, and close collaboration with analysts to deliver high-quality data products.

Qualifications

• Hands-on PySpark experience with Databricks.
• Real-time data streaming with Kafka.
• Proficiency in Airflow DAGs.
• Experience with Redshift and AWS data services.
• Strong SQL and data modeling skills.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark on Databricks
• Build and maintain real-time data streaming pipelines using Apache Kafka
• Orchestrate and schedule data workflows using Apache Airflow
• Manage and optimize data models and queries in Amazon Redshift
• Architect and deploy data solutions on AWS using services such as S3, Glue, Lambda, EMR, IAM, and CloudWatch
• Collaborate with analysts and platform teams to deliver high-quality data products
• Monitor pipeline performance and implement tuning strategies for large-scale data workloads
• Implement data quality checks, observability, and alerting across pipelines
• Participate in code reviews and contribute to engineering best practices
• Document data flows, architecture decisions, and pipeline logic

Skills

PySpark

DataFrame API

Spark SQL

Databricks

Apache Kafka

Apache Airflow

Amazon Redshift

AWS

Python

Data modeling

Query tuning

Cloud architecture

Education

Bachelor's or Master's degree in CS/Engineering or related field

Tools

Databricks

Apache Kafka

Apache Airflow

Amazon Redshift

AWS

S3

Glue

Lambda

EMR

IAM

CloudWatch

Delta Lake

dbt

Terraform

Job description

InfoSmart Technologies, Inc. is seeking a Data Engineer to design, build, and maintain scalable data pipelines using PySpark and Databricks, with a focus on streaming data via Kafka and orchestrating workflows with Airflow.

The role requires hands-on AWS data warehousing on Redshift, building cloud-native solutions with S3, Glue, Lambda, EMR, IAM, and CloudWatch, and close collaboration with analysts to deliver high-quality data products.