Databricks Developer

Tekskills Inc.; Delhi; Presencial; INR 1.800.000 - 3.200.000; A tiempo parcial

Descripción de la vacante

Tekskills Inc. is seeking a Data Engineer to architect and implement scalable data pipelines using Databricks, Spark, and Delta Lake. The role focuses on Bronze-Silver-Gold data layers, data modeling, and ensuring data quality and security across the pipeline.

You will collaborate with data scientists, analysts, and business stakeholders to deliver analytics-ready data and reliable data products in a fast-paced environment.

Formación

• Experience designing and deploying scalable data pipelines on Databricks and Spark.
• Strong Python/Scala for Spark development.
• Experience with Delta Lake and Spark DataFrame API.
• Experience migrating from Cloudera to Databricks.
• Implement data quality and security practices.

Responsabilidades

• Design and implement scalable data pipelines.
• Ingest data into Bronze, Silver, Gold with Medallion Architecture.
• Model data for analytics and business insights.
• Ensure data quality and security throughout pipelines.
• Automate, monitor, and troubleshoot pipelines.
• Collaborate with data scientists and stakeholders.

Conocimientos

Databricks Spark cert

Python/Scala for Spark

Delta Lake & DataFrame

Cloudera to Databricks migration

Data pipeline orchestration

Data quality & security

Herramientas

Databricks Jobs

Airflow

Descripción del empleo

Job Description

• Data Pipeline Design: Architect and implement scalable, reliable data pipelines using Databricks, Spark, and Delta Lake
• Medallion Architecture Implementation: Ingest raw data into the Bronze layer, process and clean it in the Silver layer, and aggregate/transform it for analytics in the Gold layer
• Data Modeling: Design and optimize data models for efficient storage, retrieval, and analytics
• Data Quality & Security: Implement data validation, quality checks, and security controls throughout the pipeline
• Automation & Monitoring: Automate pipeline execution, monitor performance, and troubleshoot issues
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality, analytics-ready data

Essential Skills

• Databricks Certified Associate Developer for Apache Spark (or equivalent)
• Strong Python/Scala for Spark development
• Experience with Delta Lake and Spark DataFrame API
• Proven experience migrating from Cloudera to Databricks
• Data pipeline orchestration (Databricks Jobs, Airflow, etc.)
• Data quality and security best practices

Desirable Skills

• Databricks Certified Data Engineer Associate
• SQL and advanced query optimization
• Cloud platforms (Azure, AWS, GCP)
• Real-time/streaming data processing (Spark Structured Streaming)
• DevOps/MLOps (CI/CD, Docker, Kubernetes)
• Experience with data governance and lineage tools