DE Architect

Tredence Inc.; Bengaluru; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

A leading data engineering firm is seeking experienced professionals in Bengaluru. This role focuses on building scalable data pipelines and optimizing big data workloads with strong expertise in Databricks, Apache Spark, and cloud-based platforms. Responsibilities include designing ETL/ELT pipelines, leading small teams, ensuring data governance compliance, and working with various cloud technologies. Candidates should have 9-12 years of experience and be capable of mentoring others. Certifications in Databricks and cloud services are preferred.

Qualifications

• 9-12 years of experience in Data Engineering.
• Expertise in building scalable data pipelines.
• Strong knowledge of cloud-based data platforms.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Databricks.
• Optimize workflows and manage Databricks clusters.
• Lead project teams and mentor junior engineers.

Skills

Databricks Platform Expertise

Apache Spark

Programming (Python/Scala)

ETL/Data Architecture

Performance Optimization

Debugging and problem-solving

Effective communication

Mentoring ability

Strategic thinking

Ownership mindset

Education

Databricks Lakehouse Fundamentals

Cloud certifications (AWS/Azure/GCP)

Job description

Experience Range: 9–12 Years
Locations: Pune, Bangalore, Chennai, Gurgaon, Hyderabad, Kolkata
Role Levels: Databricks Engineer / Associate Manager – Data Engineering / Manager – Data Engineering

Role Overview

We are looking for experienced Data Engineering professionals with strong expertise in Databricks, Apache Spark, and cloud-based data platforms. The role involves building scalable data pipelines, optimizing big-data workloads, implementing data governance, and driving best practices across Data Engineering teams.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Databricks (Python/Scala/Spark SQL).
• Work extensively on Delta Lake, including ACID transactions, schema evolution, time-travel, and performance optimizations.
• Build and optimize workflows using Databricks Workflows, DLT, and Unity Catalog.
• Configure and optimize Databricks clusters and Spark jobs for performance and cost efficiency.
• Work with cloud storage systems (S3/ADLS/GCS), IAM/RBAC, and networking/security components.
• Lead small teams in delivering Databricks-based data solutions.
• Own end-to-end design of data pipelines and distributed systems.
• Ensure data governance compliance, security controls, and best engineering practices.

For Manager – Data Engineering (10–12 Years)

All responsibilities of Associate Manager plus:

• Partner with business, product, analytics, and data science teams to align solutions with business goals.
• Lead multiple project teams, mentor engineers, and ensure delivery excellence.
• Define long-term data engineering roadmap, standards, and best practices.

Core Technical Skills (All Roles)

• Databricks Platform Expertise: Workspace, notebooks, DLT, Workflows, Unity Catalog.
• Apache Spark: PySpark, Spark SQL, Scala/Java for performance tuning.
• Programming: Python and/or Scala; SQL for analytics and data validation.
• ETL/Data Architecture: Batch & streaming pipelines, Medallion Architecture.
• Performance Optimization: Debugging data skew, memory issues, cluster tuning.
• Security & Governance: Row-level/column-level security, encryption, access controls.
• Strong analytical and problem-solving ability.
• Effective communication with stakeholders, analysts, and cross-functional teams.
• Ability to mentor juniors and enable team growth (Associate Manager/Manager).
• Strategic thinking with the ability to influence design and technology decisions.
• Ownership mindset with strong execution focus.

Preferred Certifications

• Databricks Lakehouse Fundamentals
• Cloud certifications: AWS/Azure/GCP (Associate or above)