Get a reply from this employer
Your resume and cover letter, tailored to exactly what they’re hiring for.
Acuity Knowledge Partners in Bengaluru, Karnataka, India is seeking a Senior Databricks Platform Engineer to design, build, deploy, and optimize enterprise data solutions on AWS.
The role combines advanced data engineering with platform administration, covering batch and real-time pipelines, security, and cost optimization, with governance via Unity Catalog and strong collaboration with cross-functional teams.
Assistant Director-Databricks Platform and Data Engineer ETV -SEZ1, Bengaluru, Karnataka, India 8 - 12 Years Permanent
The Senior Databricks Platform & Data Engineer will design, build, deploy, administer, and optimize enterprise data solutions using the Databricks Data Intelligence Platform on Amazon Web Services. The role combines advanced data engineering with platform administration and requires hands-on ownership of batch and realtime pipelines, platform security, Unity Catalog governance, compute optimization, deployment automation, monitoring, and production support. The engineer will work closely with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, and firewall teams to deliver secure, scalable, reliable, and cost-effective data platforms.
Databricks Platform Engineering: Advanced hands-on capability in administering, configuring, deploying, monitoring, tuning, scaling, and supporting Databricks on AWS.
Python, SQL and PySpark Engineering: Strong programming capability for developing, testing, optimizing, and troubleshooting production-grade pipelines.
Kafka and Structured Streaming: Expertise in real-time ingestion, event-driven processing, fault tolerance, checkpointing, schema evolution, latency management, and streaming recovery.
Unity Catalog, Security and Governance: Experience implementing data lineage, metadata governance, permissions, auditing, data isolation, secure sharing, and least-privilege access.
AWS Infrastructure and Networking: Working knowledge of IAM, VPCs, subnets, routing, security groups, private connectivity, S3, EC2, CloudWatch, KMS, Secrets Manager, and firewall dependencies.
Performance, Scalability and Cost Optimization: Ability to tune Spark jobs, SQL queries, streaming pipelines,
Delta tables, clusters, and serverless compute while managing cost and reliability.
Deployment and Automation: Experience with Git, CI/CD, Databricks Asset Bundles, automated testing, deployment automation, workflow orchestration, and operational monitoring.
Architecture and Trade-off Analysis: Ability to assess serverless, classic, and hybrid approaches based on security, cost, performance, scalability, supportability, and operational risk.
Production Ownership and Problem-Solving: Ability to independently diagnose and resolve complex data, platform, networking, security, streaming, and performance issues.
Cross-functional Collaboration and Communication: Ability to work effectively with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, firewall, and source-system teams.