Senior Data Engineer - Databricks

Texplorers Inc; Hyderabad; On-site; INR 1,200,000 - 2,800,000; Full time

Job summary

Texplorers Inc. is seeking a Databricks Developer to design, develop, and optimize big data solutions on Azure cloud. You will build ETL pipelines with Spark, ingest and process data from multiple sources, and tune performance while ensuring security and governance.

Ideal candidates have strong SQL skills, hands-on Databricks experience, and familiarity with CI/CD, Git, and data warehousing concepts. Collaboration with data engineers, scientists, and analysts is essential for analytics delivery.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
• Strong hands-on experience with Databricks and Apache Spark in data engineering.
• Proficiency in SQL for data analysis and transformation.
• Experience with cloud platforms (Azure, AWS, GCP) and data lakes/warehousing.
• Familiarity with CI/CD, Git, and DevOps practices for data workflows.
• Knowledge of Delta Lake, Lakehouse architecture, and performance tuning.

Responsibilities

• Design, develop, and optimize ETL pipelines in Databricks.
• Build data ingestion, transformation, and processing workflows using Spark.
• Work with structured and unstructured data from various sources.
• Optimize Spark jobs for performance, cost efficiency, and scalability.
• Implement CI/CD for Databricks workflows and version control.
• Work with Data Engineers, Data Scientists, and Business Analysts to support analytics.
• Implement security best practices and manage access controls in Databricks.
• Troubleshoot performance and scalability issues in big data processing.

Skills

Databricks

Apache Spark

PySpark/Scala/Java

SQL

Cloud platforms (Azure, AWS, GCP)

ETL pipelines

CI/CD & Git

Delta Lake / Lakehouse

Distributed computing

Education

Bachelor’s or Master’s degree in Computer Science / Data Engineering

Tools

Azure Data Lake

AWS S3

GCP BigQuery

DataPipelines

Data Warehousing

Job description

We are looking for a skilled Databricks Developer to design, develop, and optimize big data solutions using ApacheSpark and Databricks on Azure cloud. The ideal candidate will have strong experience in data engineering, ETL pipelines, and performance optimization.

Key Responsibilities:

• Design, develop, and optimize ETL pipelines in Databricks.
• Build data ingestion, transformation, and processing workflows using Apache Spark.
• Work with structured and unstructured data from various sources.
• Optimize Spark jobs for performance, cost efficiency, and scalability.
• Implement CI/CD for Databricks workflows and manage version control.
• Work closely with Data Engineers, DataScientists, and Business Analysts to support analytics and reporting.
• Implement security best practices and manage access controls in Databricks.
• Troubleshoot performance and scalability issues in big data processing.

Required Skills:

• Strong hands-on experience with Databricks and Apache Spark (PySpark/Scala/Java).
• Proficiency in SQL for data analysis and transformation.
• Experience in cloud platforms (Azure, AWS,or GCP), especially Azure Data Lake, AWS S3, or GCP BigQuery.
• Hands-on experience with ETL tools, DataPipelines, and Data Warehousing.
• Strong knowledge of CI/CD, Git, and DevOpstools for data workflows.
• Knowledge of Delta Lake, Lakehouse architecture, and performance tuning.
• Experience in handling large-scale datasets and distributed computing.

Preferred Skills:

• Experience with Databricks MLflow and machine learning pipelines.
• Knowledge of Kafka, Airflow, or Snowflake.
• Understanding of data governance,compliance, and security policies.

Education & Experience:

• Bachelor’s or Master’s degree in ComputerScience, Data Engineering, or a related field.