Senior Data Platform Engineer - Azure Databricks Lakehouse

Solvimate Pvt. Ltd.

Cary (NC)

Ibrido

USD 140.000 - 145.000

Tempo pieno

3 giorni fa

Candidati tra i primi

Generatore di candidature

Ricevi una risposta da questo datore di lavoro — un curriculum e una lettera di presentazione personalizzati, che corrispondono esattamente a ciò che sta cercando.

Supera i filtri ATS

Descrizione del lavoro

Solvimate Pvt. Ltd. is building an enterprise-scale data and AI platform with a hybrid role in Cary, NC.

You will shape the Lakehouse foundation using Azure Databricks and related Azure tools, delivering end-to-end data engineering solutions and AI-assisted capabilities. You will design and deploy production pipelines, governance, and scalable architectures, while mentoring engineers and collaborating with stakeholders.

Competenze

12+ years of professional experience in Data Engineering or related fields.
Expert-level proficiency in Python, Scala, and PySpark.
Strong SQL and data modeling experience, including dimensional and normalized models.
Extensive hands-on experience with Azure Databricks and Delta Lake.
Experience with Databricks Jobs & Workflows, Unity Catalog, cluster management and performance tuning.
Hands-on Azure Data Factory and metadata-driven pipeline frameworks.
Experience with Azure Event Hubs, Kafka, or Spark Structured Streaming.
Proven experience designing enterprise-scale Lakehouse/Medallion architectures.
At least 3 years of hands-on experience designing and deploying production LLM/AI systems.
Strong understanding of RAG, embeddings, vector/hybrid retrieval, prompt engineering, and agentic workflows.
Experience with LangChain, LlamaIndex, or LangGraph.
Experience with Azure OpenAI, OpenAI, or Databricks Model Serving.
Experience implementing AI evaluation frameworks including golden datasets, regression testing, and human-in-the-loop validation.
Strong understanding of Azure security: Entra ID, RBAC, Managed Identities, Key Vault, ACLs, Private Endpoints.

Mansioni

Design and implement scalable Lakehouse/Medallion Architecture (Bronze, Silver, Gold).
Develop production-grade pipelines using Python, Scala, PySpark, Spark Structured Streaming, Delta Lake.
Build metadata-driven ingestion frameworks for batch, CDC, and streaming sources.
Work across Azure Databricks, ADLS Gen2, Azure Data Factory, Azure Event Hubs, and Kafka.
Design Delta Lake tables with partitioning, schema evolution, and data retention; enforce data contracts.
Optimize Spark workloads, clusters, pipelines, and overall platform performance.
Establish engineering standards: coding practices, automated testing, code/PR reviews.
Implement CI/CD with Azure DevOps, Terraform, and Databricks Asset Bundles.
Develop AI-assisted ingestion, canonical mapping, data quality, reconciliation workflows.
Design production-grade LLM, RAG, and agentic AI workflows including embeddings and vector search.
Build semantic-layer and text-to-SQL capabilities for enterprise data access.
Implement data governance, lineage, access control, and PII protection using Unity Catalog.
Monitor data quality, schema drift, anomalies, reconciliation issues, and pipeline health.
Mentor data engineers and present designs to stakeholders.