Senior/Lead Data Engineer

Vibotek LLC; Alpharetta (GA); Hybrid; USD 120,000 - 150,000; Full time

Job summary

A leading technology firm located in Alpharetta is looking for an experienced Data Engineer with over 8 years of hands-on experience in enterprise environments. The ideal candidate should have strong expertise in Azure services and proficiency in Apache Spark with Python. Responsibilities include designing end-to-end data solutions and working with cross-functional teams. This position offers a hybrid work model with the possibility for immediate engagement.

Qualifications

• 8+ years of hands-on data engineering experience in enterprise environments.
• Strong expertise in Azure services, especially Azure Databricks.
• Advanced proficiency in Apache Spark with Python (PySpark).

Skills

Data engineering experience

Azure services (Databricks, Functions, Data Factory)

Apache Spark with Python (PySpark)

SQL and query optimization

ETL/ELT methodologies

Delta Lake

Data modelling

Batch and real-time processing

Data architecture principles

Problem-solving skills

Collaboration with cross-functional teams

Tools

Azure DevOps

Git

Terraform

Azure Purview

Job description

Location

Alpharetta

Required Citizenship / Work Permit / Visa Status

US Citizens, green card holder

Notice period

0 to 15 days only

Job stability

Job stability is mandatory

Interview Process

2 Technical rounds + 1 Client round. 3 days in office, Hybrid model.

What You Need

• 8+ years of hands-on data engineering experience in enterprise environments.
• Strong expertise in Azure services, especially Azure Databricks, Functions, and Azure Data Factory (preferred).
• Advanced proficiency in Apache Spark with Python (PySpark).
• Strong command over SQL, query optimization, and performance tuning.
• Deep understanding of ETL/ELT methodologies, data pipelines, and scheduling/orchestration.
• Hands‑on experience with Delta Lake (ACID transactions, optimization, schema evolution).
• Strong experience in data modelling (normalized, dimensional, lakehouse modelling).
• Experience in both batch processing and real-time/streaming data (Kafka, Event Hub, or similar).
• Solid understanding of data architecture principles, distributed systems, and cloud-native design patterns.
• Ability to design end‑to‑end solutions, evaluate trade‑offs, and recommend best‑fit architectures.
• Strong analytical, problem‑solving, and communication skills.
• Ability to collaborate with cross‑functional teams and lead technical discussions.

Core skills include Java, Python, and Apache Spark.

Preferred Skills

• Experience with CI/CD tools such as Azure DevOps and Git.
• Familiarity with IaC tools (Terraform, ARM).
• Exposure to data governance and cataloging tools (Azure Purview).
• Experience supporting machine learning or BI workloads on Databricks.