Lead Full Stack Engineer

InfoBeans; Charlotte (NC); On-site; USD 140,000 - 200,000; Full time

Job summary

InfoBeans in Charlotte, NC is seeking a Senior / Lead Data Engineer to design, build, and optimize enterprise-scale data platforms. The role requires deep hands-on expertise with Azure Databricks, PySpark, ADF, and Delta Lake to deliver reliable, scalable solutions.

You will work closely with architects, analysts, and business stakeholders to implement best practices in data governance, streaming pipelines, and lakehouse architectures, while driving performance and cost efficiency.

Qualifications

• 10+ years of experience designing enterprise data platforms.
• Hands-on expertise with Azure Databricks, PySpark, ADF, Delta Lake.
• Experience building scalable, reliable data pipelines and governance.

Responsibilities

• Design, build, and optimize enterprise-scale data platforms.
• Collaborate with architects, analysts, and stakeholders.
• Deliver high-performance, scalable data solutions.
• Lead data engineering best practices and CI/CD automation.

Skills

Azure Databricks

PySpark

ADF (Azure Data Factory)

Delta Lake

Data warehousing

Streaming

CI/CD

Spark tuning

Data governance

Team collaboration

Tools

Databricks

Azure Data Factory

Delta Lake

Job description

Job Description:

InfoBeans is hiring an experienced Senior / Lead Data Engineer to design, build, and optimize enterprise‑scale data platforms. The ideal candidate has strong hands‑on expertise with Azure Databricks, PySpark, ADF, Delta Lake, and modern data warehousing. In this role, you will work closely with architects, analysts, and business stakeholders to deliver high‑performance, reliable, and scalable data solutions.

If you thrive in fast‑paced environments and enjoy solving complex data challenges, we’d love to connect.

Experience: 10+ years

Contract: C2H / W2 / 1099

Duration: 12 months

Advanced Spark Optimization - Optimizing DataFrame transformations to avoid shuffles and wide dependencies, Adaptive Query Execution (AQE) tuning, Handling skew, Caching & checkpointing strategies, Cluster sizing & autoscaling strategies

Databricks Workflows & Orchestration - Orchestrating ETL/ELT with Jobs & Workflows, Multi-task job pipelines (with task dependencies), Job clusters vs all-purpose clusters, Error handling & retries, CI/CD deployment using repos

Unity Catalog & Enterprise Data Governance - Fine‑grained access control (schemas, tables, columns, views), Data lineage tracking, Secure data sharing across workspaces, Managing tokens, service principals & permissions models

Databricks SQL & Lakehouse Warehouses - Writing high-performance queries on the Lakehouse, Materialized views, SQL dashboards, Understanding Photon execution engine for high-speed queries

Streaming & Real‑Time Pipelines - Structured Streaming internals, Auto Loader for incremental ingest, Trigger types & watermarking, Handling late‑arriving data, Stateful stream processing

Advanced Cluster & Compute Management - Spot vs on‑… Photon runtime vs standard, Cluster policies for cost control, SQL warehouses vs all-purpose compute, Monitoring with Ganglia / metrics dashboards

Best Practices in Lakehouse Architecture - Medallion Architecture patterns, Modular, reusable ETL code patterns, Cost optimization strategies, Data quality frameworks (e.g., expectations, constraints)

DevOps & CI/CD Integration - Git integration via Databricks Repos, Promoting code across dev/test/prod, Using the Databricks CLI, Automated deployments using Azure DevOps/GitHub Actions

Nice to have:

Azure Data Factory - Understanding of Pipelines, Activities, and Datasets, Linked Services configuration, Integration Runtime types (Auto‑IR, Self‑Hosted IR, Azure IR), Source and Sink concepts in ADF