Databricks Developer

Tekskills Inc.; Hyderabad; Presencial; INR 1.500.000 - 2.100.000; Jornada completa

Descripción de la vacante

Tekskills Inc. is seeking a data engineering professional to design and implement scalable data pipelines using Databricks, Spark, and Delta Lake. You will drive the Medallion architecture across Bronze, Silver, and Gold layers for analytics.

You will collaborate with data scientists and business stakeholders to deliver clean, analytics-ready data, ensure data quality and security, and automate pipelines with robust monitoring and troubleshooting capabilities.

Formación

• Databricks Certified Associate Developer for Apache Spark or equivalent required.
• Strong Python/Scala for Spark development.
• Experience with Delta Lake and Spark DataFrame API.
• Proven experience migrating from Cloudera to Databricks.
• Data pipeline orchestration with Databricks Jobs or Airflow.
• Data quality and security best practices are essential.

Responsabilidades

• Design and implement scalable data pipelines using Databricks, Spark, and Delta Lake.
• Architect Bronze-Silver-Gold medallion data architecture for analytics.
• Model data for efficient storage, retrieval, and analytics.
• Implement data quality checks and security controls throughout pipelines.
• Automate pipeline execution and monitor performance.
• Collaborate with data scientists, analysts, and stakeholders on analytics-ready data.

Conocimientos

Databricks Certified Associate Dev for

Python/Scala for Spark

Delta Lake & Spark DataFrame

Migration Cloudera→Databricks

Data pipeline orchestration

Data quality & security

Herramientas

Databricks

Airflow

Spark

Delta Lake

Descripción del empleo

Job Description

• Data Pipeline Design: Architect and implement scalable, reliable data pipelines using Databricks, Spark, and Delta Lake
• Medallion Architecture Implementation: Ingest raw data into the Bronze layer, process and clean it in the Silver layer, and aggregate/transform it for analytics in the Gold layer
• Data Modeling: Design and optimize data models for efficient storage, retrieval, and analytics
• Data Quality & Security: Implement data validation, quality checks, and security controls throughout the pipeline
• Automation & Monitoring: Automate pipeline execution, monitor performance, and troubleshoot issues
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality, analytics-ready data

Essential Skills

• Databricks Certified Associate Developer for Apache Spark (or equivalent)
• Strong Python/Scala for Spark development
• Experience with Delta Lake and Spark DataFrame API
• Proven experience migrating from Cloudera to Databricks
• Data pipeline orchestration (Databricks Jobs, Airflow, etc.)
• Data quality and security best practices

Desirable Skills

• Databricks Certified Data Engineer Associate
• SQL and advanced query optimization
• Cloud platforms (Azure, AWS, GCP)
• Real-time/streaming data processing (Spark Structured Streaming)
• DevOps/MLOps (CI/CD, Docker, Kubernetes)
• Experience with data governance and lineage tools