Assistant Director-Databricks Platform and Data Engineer

Acuity Knowledge Partners

Bengaluru

On-site

INR 1,800,000 - 2,400,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Acuity Knowledge Partners in Bengaluru, Karnataka, India is seeking a Senior Databricks Platform Engineer to design, build, deploy, and optimize enterprise data solutions on AWS.

The role combines advanced data engineering with platform administration, covering batch and real-time pipelines, security, and cost optimization, with governance via Unity Catalog and strong collaboration with cross-functional teams.

Qualifications

  • 8–12 years of experience in data engineering and platform engineering.
  • Hands-on with Databricks on AWS and production data pipelines.
  • Strong knowledge of Spark, PySpark, SQL and Delta Lake.

Responsibilities

  • Design, build, and maintain batch and real-time data pipelines on Databricks and Spark.
  • Administer Databricks on AWS, including workspaces, permissions, and jobs.
  • Tune Spark workloads and optimize compute, storage, and streaming latency.
  • Implement Unity Catalog governance and secure data sharing.

Skills

Data engineering

Tools

Databricks
Python
SQL
PySpark
Delta Lake
Apache Kafka
Spark Structured Streaming
AWS
Unity Catalog
Kubernetes

Job description

Assistant Director-Databricks Platform and Data Engineer ETV -SEZ1, Bengaluru, Karnataka, India 8 - 12 Years Permanent

Job Description
Job purpose

The Senior Databricks Platform & Data Engineer will design, build, deploy, administer, and optimize enterprise data solutions using the Databricks Data Intelligence Platform on Amazon Web Services. The role combines advanced data engineering with platform administration and requires hands-on ownership of batch and realtime pipelines, platform security, Unity Catalog governance, compute optimization, deployment automation, monitoring, and production support. The engineer will work closely with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, and firewall teams to deliver secure, scalable, reliable, and cost-effective data platforms.

Key responsibilities
  • Design, build, and maintain production-grade batch and real-time data pipelines using Databricks, Python, SQL, PySpark, Delta Lake, Apache Kafka, and Spark Structured Streaming.
  • Administer, configure, deploy, monitor, and support Databricks environments on AWS, including workspaces, compute resources, permissions, jobs, and environment-specific configurations.
  • Tune and scale Spark workloads, SQL queries, Delta tables, streaming jobs, and compute resources to meet performance, reliability, latency, and cost objectives.
  • Evaluate serverless, classic, and hybrid compute models, recommending the appropriate approach based on workload characteristics, security, cost, scalability, performance, and operational complexity.
  • Build fault-tolerant streaming pipelines using Kafka and Structured Streaming, including checkpointing, schema evolution, failure recovery, latency monitoring, and streaming-performance optimization.
  • Implement Unity Catalog for centralized governance, metadata management, lineage, data discovery, access control, auditing, and secure data sharing.
  • Automate data ingestion, transformation, testing, deployment, monitoring, and recovery using Databricks workflows, Lakeflow Jobs, Auto Loader, Git-based CI/CD, and Databricks Asset Bundles where applicable.
  • Establish secure connectivity between source systems, AWS services, Kafka, and Databricks, working with infrastructure, network, cybersecurity, and firewall teams.
  • Configure and troubleshoot AWS networking and security, including VPCs, subnets, routing, security groups, private connectivity, IAM roles and policies, encryption keys, and secrets.
  • Integrate Databricks with AWS services such as Amazon S3, EC2, CloudWatch, KMS, and Secrets Manager, with exposure to Kubernetes and Amazon EKS where applicable.
  • Diagnose and resolve production issues involving source availability, data ingestion, streaming latency, Spark performance, compute capacity, permissions, networking, firewalls, and platform security.
  • Document architecture decisions and technical trade-offs, clearly explaining the impact of design choices on cost, security, performance, availability, scalability, and maintainability.

Databricks Platform Engineering: Advanced hands-on capability in administering, configuring, deploying, monitoring, tuning, scaling, and supporting Databricks on AWS.

Python, SQL and PySpark Engineering: Strong programming capability for developing, testing, optimizing, and troubleshooting production-grade pipelines.

Kafka and Structured Streaming: Expertise in real-time ingestion, event-driven processing, fault tolerance, checkpointing, schema evolution, latency management, and streaming recovery.

Unity Catalog, Security and Governance: Experience implementing data lineage, metadata governance, permissions, auditing, data isolation, secure sharing, and least-privilege access.

AWS Infrastructure and Networking: Working knowledge of IAM, VPCs, subnets, routing, security groups, private connectivity, S3, EC2, CloudWatch, KMS, Secrets Manager, and firewall dependencies.

Performance, Scalability and Cost Optimization: Ability to tune Spark jobs, SQL queries, streaming pipelines,

Delta tables, clusters, and serverless compute while managing cost and reliability.

Deployment and Automation: Experience with Git, CI/CD, Databricks Asset Bundles, automated testing, deployment automation, workflow orchestration, and operational monitoring.

Architecture and Trade-off Analysis: Ability to assess serverless, classic, and hybrid approaches based on security, cost, performance, scalability, supportability, and operational risk.

Production Ownership and Problem-Solving: Ability to independently diagnose and resolve complex data, platform, networking, security, streaming, and performance issues.

Cross-functional Collaboration and Communication: Ability to work effectively with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, firewall, and source-system teams.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Platform Engineer
Databricks Platform Engineer

Zorba Consulting India • Mumbai

On-site
INR 2,500,000 - 4,200,000
Senior Data Engineer
Senior Data Engineer

Wissen Technology • Bengaluru

Hybrid
INR 3,000,000 - 5,400,000
Databricks Administrator / Databricks Platform Engineer
Databricks Administrator / Databricks Platform Engineer

VMC Soft Technologies • Kolkata District

On-site
INR 2,400,000 - 4,000,000
Data Bricks Administrator
Data Bricks Administrator

Nisum • Hyderabad

On-site
INR 1,800,000 - 3,200,000
Databricks - Data Engineer
Databricks - Data Engineer

Tredence Inc. • Bengaluru

On-site
INR 1,500,000 - 2,500,000
Senior Databricks Platform Engineer
Senior Databricks Platform Engineer

Wipro Ltd. • Bengaluru, Mumbai, Chennai

On-site
INR 4,000,000 - 7,000,000
Databricks Lead
Databricks Lead

Ascendion • Bengaluru

Hybrid
INR 400,000 - 700,000
Hybrid work in Bengaluru
Relocation allowed
Databricks Engineer
Databricks Engineer

Fa Ewjt Saasfaprod1 • Chennai

On-site
INR 3,500,000 - 6,000,000
Databricks Engineer
Databricks Engineer

Infosys • Bengaluru

On-site
INR 2,800,000 - 4,400,000
Senior Manager
Senior Manager

ExlService Holdings, Inc. • Bengaluru

Hybrid
INR 2,400,000 - 3,600,000