Senior Data Engineer

Texplorers Inc; Hyderabad; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading tech company in Hyderabad is looking for an experienced Data Engineer to develop and optimize ETL pipelines using Apache Spark in cloud environments. The ideal candidate should have 5-7 years of experience, particularly within the Retail sector, and a strong command of SQL and Databricks. Responsibilities include managing complex data lifecycles and collaborating on data-driven solutions. This full-time role emphasizes innovation and continuous improvement in data engineering practices.

Qualifications

• 5–7 years of specialized experience in Data Engineering and big data architecture.
• Experience in the Retail sector is mandatory.

Responsibilities

• Design and optimize Databricks ETL pipelines for high-quality data delivery.
• Develop scalable data ingestion workflows with Apache Spark.
• Manage complex data lifecycles involving structured and unstructured datasets.
• Fine-tune Spark jobs for performance and cost-efficiency.
• Establish CI/CD patterns for Databricks workflows.
• Collaborate with stakeholders for technical solutions.
• Enforce security protocols in the Databricks environment.
• Diagnose and resolve big data processing bottlenecks.

Skills

Databricks development

Apache Spark

SQL

Cloud ecosystems (Azure, AWS, GCP)

ETL/ELT pipelines

CI/CD pipelines

Git version control

Delta Lake

Distributed computing

Education

Bachelor's or Master's degree in Computer Science or related field

Tools

Apache Kafka

Airflow

Snowflake

Job description

Texplorers Inc. is a leader in digital transformation for global enterprises, with over 15 years of experience in cloud technology and AI expertise. The company focuses on empowering businesses to scale efficiently and continuously improve through their innovation ecosystem. Texplorers Inc. is committed to innovation, sustainability, and inclusion, ensuring perpetual learning and growth for their clients.

We are seeking candidates who can join us immediately.

We are seeking an experienced Data Engineer to build and refine robust big data solutions. You will be responsible for designing high-performance ETL pipelines using Apache Spark across major cloud environments (Azure, AWS, or GCP), with a heavy focus on data engineering excellence and system optimization.

Key Responsibilities:

• Lead the end-to-end design and optimization of Databricks ETL pipelines to ensure high-quality data delivery.
• Develop scalable data ingestion and transformation workflows by leveraging the full power of the Apache Spark framework.
• Manage complex data lifecycles involving both structured and unstructured datasets from diverse originating sources.
• Fine-tune Spark jobs to balance peak performance with cost-efficiency and long-term scalability.
• Establish robust CI/CD patterns and version control standards for all Databricks-based workflows.
• Collaborate with stakeholders across engineering, data science, and analytics to translate business needs into technical solutions.
• Enforce rigorous security protocols and access management policies within the Databricks environment.
• Diagnose and resolve complex bottlenecks related to big data processing and system scalability.

Required Skills:

• Proven track record of developing high-performance solutions using Databricks and Apache Spark (proficient in PySpark, Scala, or Java).
• Expert-level command of SQL for complex data modeling, transformation, and analytical querying.
• Deep familiarity with cloud ecosystems (Azure, AWS, or GCP), with specific expertise in storage and warehousing solutions like ADLS, S3, or BigQuery.
• Significant experience building end-to-end ETL/ELT pipelines and managing scalable Data Warehousing environments.
• Strong grip on CI/CD pipelines, Git version control, and DevOps methodologies tailored for data engineering.
• In-depth knowledge of Delta Lake and Lakehouse architecture, with a focus on advanced performance tuning and optimization.
• Skilled in managing massive-scale datasets within distributed computing environments.

Preferred Skills:

• Skilled in leveraging Databricks MLflow to track, deploy, and scale machine learning models.
• Familiarity with modern data stack components such as Apache Kafka for streaming, Airflow for workflow automation, or Snowflake for cloud warehousing.
• Deep understanding of data privacy standards, access control policies, and corporate compliance requirements.

Education & Experience:

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a strictly related technical discipline.
• 5–7 years of specialized experience in Data Engineer and big data architecture.
• Mandatory background in the Retail sector, with a deep understanding of industry-specific data challenges.

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• Information Technology
• IT Services and IT Consulting