Senior Data Engineer - Databricks

DataNimbus Inc.; Hyderabad; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

DataNimbus Inc. is seeking an experienced Data Engineer with Databricks expertise in Hyderabad to lead complex customer projects and build scalable data architectures.

You will design end-to-end data solutions, migrate data on Databricks, and mentor junior engineers while collaborating with customers and engagement managers.

Qualifications

• 8+ years of experience in data engineering, data platforms, and analytics.
• Hands-on experience in Databricks is a must.
• Should have Databricks Certification at the professional level.
• Experience with large-scale data migrations to/from Databricks.
• Consulting / customer-facing experience with external clients.
• Comfortable writing code in Python or Scala.
• Working knowledge of two or more cloud ecosystems (AWS, Azure, GCP).
• Deep experience with distributed computing with Apache Spark.

Responsibilities

• Handle a variety of customer-technical projects, including reference architectures and how-tos.
• Scope professional services with engagement managers, guided by customer input.
• Lead transformational big data projects, end-to-end design, build and deployment of AI apps.
• Consult on architecture and design; bootstrap or implement customer projects.
• Provide escalated support for customer operational issues.
• Collaborate with project manager, architect, and customer teams to deliver.
• Mentor and guide junior data engineers and team members.

Skills

Databricks

Python

Scala

Spark

CI/CD

MLOps

Cloud (AWS/Azure/GCP)

End-to-end data architectures

Education

Databricks Certification (Professional)

Job description

Responsibilities:

• Handle a variety of impactful customer-technical projects that may include designing and building reference architectures, creating how-tos, and operationalizing customer use cases.
• Work with engagement managers to scope a variety of professional services and work with input from the customer.
• Guide strategic customers as they implement transformational big data projects and 3rd-party migrations, including end-to-end design, build, and deployment of industry-leading big data and AI applications.
• Consult on architecture and design; bootstrap or implement customer projects, which leads to a customer's successful understanding, evaluation, and adoption of Databricks.
• Provide an escalated level of support for customer operational issues.
• Collaborate with the project manager, architect, and customer teams to ensure the technical components of the engagement are delivered to meet customer needs.
• Mentor and guide junior data engineers and team members.

Requirements:

• 8+ years of experience in data engineering, data platforms, and analytics.
• Hands-on experience in Databricks is a must.
• Should have Databricks Certification at the professional level.
• Experience with large-scale data migrations to/from Databricks.
• Consulting / customer-facing experience, working with external clients across a variety of industry markets.
• Comfortable writing code in either Python or Scala.
• Working knowledge of two or more common cloud ecosystems (AWS, Azure, GCP) with expertise in at least one.
• Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals.
• Familiarity with CI/CD for production deployments.
• Working knowledge of MLOps.
• Design and deployment of performant end-to-end data architectures.
• Experience with technical project delivery, managing scope and timelines.
• Documentation and whiteboarding skills.
• Experience working with clients and managing conflicts.
• Deployment and integration of Databricks-based solutions to complete customer projects.

Must-have Databricks certification Professional; non-negotiable:

• Strong communication and collaboration skills.
• Excellent problem-solving skills.
• Ability to travel up to 30% when needed.