Lead I - Data Engineering

UST

Pune

Vor Ort

INR 2.500.000 - 5.000.000

Vollzeit

vor 31 Stunden
Sei unter den ersten Bewerbenden

Erhalte eine Antwort von diesem Arbeitgeber

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Zusammenfassung

UST in Pune seeks a Senior Data Engineer (Azure Databricks) to build scalable batch and streaming pipelines using Spark and PySpark. The role focuses on Databricks on Azure, Delta Lake governance, and SQL-driven data modeling for modern lakehouse architectures.

The ideal candidate will implement robust data pipelines, optimize performance, and collaborate on CI/CD for data platforms while ensuring security and governance standards.

Qualifikationen

  • Extensive experience with Spark and PySpark in batch/streaming pipelines.
  • Deep knowledge of Databricks on Azure including Unity Catalog concepts.
  • Strong SQL skills and data modeling for lakehouse patterns.
  • Proficiency in Python, data quality, and CI/CD practices for data platforms.
  • Familiarity with Delta Lake features and governance.

Kenntnisse

Spark & PySpark
Azure Databricks
SQL
Python
Data modeling
Data lake architectures

Tools

Delta Lake
Power BI
dbt
Kafka/Event Hubs
Terraform
Airflow
Azure Data Factory

Jobbeschreibung

Role Description

Required skills for Senior Data Engineer (Azure Databricks)

  • Spark and PySpark ‣ 5+ years building batch and/or streaming pipelines with Spark ‣ Deep understanding of Spark fundamentals: lazy evaluation, transformations vs. actions, parti tions, shuffles, joins, and caching ‣ (Nice to have) Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization ‣ (Nice to have) Structured Streaming and Auto Loader for incremental ingestion
  • Databricks platform on Azure ‣ Unity Catalog: catalogs/schemas/tables, permissions, external locations, storage credentials, lineage ‣ Databricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters ‣ Git integration (Repos), CI/CD, and environment promotion ‣ (Nice to have) Declarative pipelines with Delta Live Tables (DLT): expectations, CDC, quality rules, event logs
  • Cloud storage and access (Azure) ‣ Secret management via Databricks secret scopes and Key Vault integration ‣ (Nice to have) ADLS Gen2: ABFS access, ACLs vs. RBAC, mounts vs. direct access, OAuth/ service principals/managed identity
  • Delta Lake ‣ ACID transactions, time travel, MERGE/UPDATE/DELETE ‣ Schema enforcement and evolution ‣ (Nice to have) OPTIMIZE, ZORDER, VACUUM and retention policies ‣ (Nice to have) Compaction and smallfiles mitigation strategies
  • SQL and data querying ‣ Complex SQL with CTEs and window functions ‣ (Nice to have) Query optimization and execution plan analysis in Databricks SQL
  • Data modeling and architecture ‣ Medallion (bronze/silver/gold) patterns for data lakes ‣ Partitioning and file layout to match read patterns ‣ Designing tables for CDC and incremental processing ‣ (Nice to have) Lakehouse design tradeoffs and table constraints
  • Largescale data processing ‣ Cluster sizing, autoscaling, and cost optimization (spot, pools) ‣ Observability: Spark UI, metrics, event logs, Delta logs ‣ (Nice to have) Handling TBscale datasets, throughput vs. latency tradeoffs
  • Governance, security, and compliance ‣ (Nice to have) Privilege model in Unity Catalog; table, row, and columnlevel security patterns ‣ (Nice to have) PII handling, masking/anonymization, auditability, lineage
  • Software engineering practices ‣ Python packaging, code quality, testing (pytest, chispa), code reviews ‣ CI/CD for Databricks (Repos, asset bundles/wheels), artifact versioning ‣ Documentation and stakeholder communication
  • Nice to have (optional) ‣ Scala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns
Skills

PySpark, Azure Data Factory, SQL, Data Lakes

Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Lead Data Architect (Azure Databricks)
Lead Data Architect (Azure Databricks)

Diacto Technologies • Indien

Remote
INR 4.000.000 - 7.000.000
Lead I - Azure Data Engineer FS
Lead I - Azure Data Engineer FS

UST • Maharashtra

Vor Ort
INR 1.500.000 - 2.500.000
Azure Data Engineer
Azure Data Engineer

InfoBeans • Indore District, Pune District

Vor Ort
INR 3.000.000 - 5.400.000
Senior Databricks Engineer
Senior Databricks Engineer

Value Vision • Bengaluru

Vor Ort
INR 4.200.000 - 6.600.000
Databricks Lead Job ID: 411651
Databricks Lead Job ID: 411651

Blumetra Solutions LLC • Hyderabad

Hybrid
INR 1.200.000 - 1.900.000
Senior Databricks & Governance Engineer
Senior Databricks & Governance Engineer

Papigen • Indien

Vor Ort
INR 3.000.000 - 5.500.000
Azure Databricks Engineer
Azure Databricks Engineer

Turnkeylearning • Hyderabad

Vor Ort
INR 1.000.000 - 2.000.000
Databricks Lead
Databricks Lead

Blumetra Solutions LLC • Hyderabad

Hybrid
INR 4.000.000 - 7.000.000
Data Engineer
Data Engineer

Calidad Infotech • Ahmedabad

Vor Ort
INR 1.600.000 - 2.400.000
Data Engineer (Azure & Databricks)
Data Engineer (Azure & Databricks)

Lufthansa Technik Services India • Bengaluru

Vor Ort
INR 1.200.000 - 1.500.000