Senior Data Engineer - Databricks

DataNimbus; Hyderabad; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

DataNimbus in Hyderabad is seeking a senior data engineer to drive customer-facing projects on the Databricks platform, including end-to-end design, build, and deployment of big data and AI solutions.

You will collaborate with engagement managers, guide strategic customers through migrations, and provide expert architecture consulting, while mentoring junior engineers and ensuring successful project delivery.

Qualifications

• 6+ years of experience in data engineering, data platforms, and analytics on the Databricks platform.

Responsibilities

• Handle a variety of impactful customer-technical projects that may include designing and building reference architectures, creating how-tos, and operationalizing customer use cases.
• Work with engagement managers to scope a variety of professional services and work with input from the customer.
• Guide strategic customers as they implement transformational big data projects and 3rd-party migrations, including end-to-end design, build, and deployment of industry-leading big data and AI applications.
• Consult on architecture and design; bootstrap or implement customer projects, which leads to a customer's successful understanding, evaluation, and adoption of Databricks.
• Provide an escalated level of support for customer operational issues.
• Collaborate with the project manager, architect, and customer teams to ensure the technical components of the engagement are delivered to meet customer needs.
• Mentor and provide guidance to junior data engineers and team members.

Skills

Python

Scala

Databricks

Spark

CI/CD

MLOps

Big data

Data engineering

Documentation

Communication

Tools

AWS

Azure

GCP

Job description

Responsibilities:

• Handle a variety of impactful customer-technical projects that may include designing and building reference architectures, creating how-tos, and operationalizing customer use cases.
• Work with engagement managers to scope a variety of professional services and work with input from the customer.
• Guide strategic customers as they implement transformational big data projects and 3rd-party migrations, including end-to-end design, build, and deployment of industry-leading big data and AI applications.
• Consult on architecture and design; bootstrap or implement customer projects, which leads to a customer's successful understanding, evaluation, and adoption of Databricks.
• Provide an escalated level of support for customer operational issues.
• Collaborate with the project manager, architect, and customer teams to ensure the technical components of the engagement are delivered to meet customer needs.
• Mentor and provide guidance to junior data engineers and team members.

Requirements:

• 6+ years of experience in data engineering, data platforms, and analytics on the Databricks platform.
• Experience with large-scale data migrations to/from Databricks.
• Consulting / customer-facing experience, working with external clients across a variety of industry markets.
• Comfortable writing code in either Python or Scala.
• Working knowledge of two or more common cloud ecosystems (AWS, Azure, GCP) with expertise in at least one.
• Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals.
• Familiarity with CI/CD for production deployments.
• Working knowledge of MLOps.
• Design and deployment of performant end-to-end data architectures.
• Experience with technical project delivery, managing scope and timelines.
• Documentation and white-boarding skills.
• Experience working with clients and managing conflicts.
• Deployment and integration of Databricks-based solutions to complete customer projects.

Must-have Databricks certifications; non-negotiable:

• Strong communication and collaboration skills.
• Excellent problem-solving skills.
• Ability to travel up to 30% when needed.