Lead Data Engineer

HiLabs; Bethesda (MD); On-site; USD 120,000 - 150,000; Full time

Benefits offered by this job

Competitive base salary

Comprehensive benefits

401k

PTOs

Relocation support

Job summary

A cutting-edge healthcare data platform company is seeking a Lead Data Engineer with expertise in Big Data technologies. This role involves leading the end-to-end design of data platforms, and building scalable systems using Spark and PySpark. Candidates should have 6 to 10 years of experience in large-scale architecture and distributed systems. The position is onsite in Bethesda, Maryland, offering competitive salary and comprehensive benefits, along with opportunities for mentorship and growth.

Qualifications

• 6 to 10 years of hands-on experience building and scaling Big Data applications.
• Strong expertise in Spark and PySpark.
• Experience with big data ecosystem tools like Hive and Solr.
• Solid understanding of large-scale architecture.

Responsibilities

• Lead end-to-end design and development of Big Data platforms.
• Architect and build high-performance distributed data systems.
• Collaborate closely with product, analytics, and DevOps teams.
• Mentor engineers and drive coding standards.

Skills

Spark

PySpark

Distributed systems

Data engineering

Workflow orchestration tools

Relational databases

AWS

Communication

Education

Bachelor’s or Master’s degree in Computer Science or related discipline

Tools

Apache Solr

Hive

Elasticsearch

Airflow

CI/CD tools

Job description

HiLabs is looking for highly motivated and technically strong Lead Data Engineers with deep expertise in Big Data platforms and a passion for building scalable, data-intensive systems. The ideal candidates will have strong hands-on experience in Spark, PySpark, distributed systems, and modern data ecosystem tools, and will enjoy owning complex data platforms from design to production.

The individuals who will join the HiLabs engineering team should be continually striving to advance engineering excellence, platform scalability, and data innovation. The mission is to power the next generation of healthcare intelligence platforms through innovation, collaboration, and transparency. You will be a leader and a doer who thrives in a fast-paced, onsite, product-driven environment.

Responsibilities

• Lead end-to-end design and development of Big Data and backend platforms
• Architect and build scalable, secure, and high-performance distributed data systems
• Design and implement robust Spark and PySpark-based data pipelines
• Develop modular, reusable, and scalable components aligned with business needs
• Manage the complete software development lifecycle from design through deployment
• Collaborate closely with product, analytics, and DevOps teams
• Mentor engineers and drive high coding standards and best practices
• Perform code reviews, testing, debugging, and performance optimization
• Ensure data platform reliability, scalability, and operational excellence
• Drive continuous improvement in data engineering architecture and tooling
• Implement CI/CD processes and automation for data workflows
• Ensure adherence to security, compliance, and high-quality engineering standards

Desired Profile

• Bachelor’s or Master’s degree in Computer Science, Mathematics, or a related quantitative discipline from a reputed institution
• Graduates from Tier-1 institutes such as IIT, NIT, or BITS are strongly preferred
• US Master’s degree is a strong advantage; recent graduates from 2023, 2024, or 2025 with strong relevant experience are encouraged to apply
• 6 to 10 years of hands-on experience building and scaling Big Data applications and distributed data platforms
• Strong expertise in Spark and PySpark
• Experience with big data ecosystem tools such as Apache Solr, Hive, HBase
• Experience with Elasticsearch and MongoDB
• Hands-on experience with workflow orchestration tools such as Airflow or Oozie
• Strong experience with relational databases such as MySQL, SQL Server, or Oracle
• Solid understanding of distributed systems and large-scale architecture
• Experience with version control tools such as Git or Bitbucket
• Experience with CI/CD tools such as Maven, Jenkins, and JIRA
• Experience working in Agile software delivery environments
• Strong debugging, performance tuning, and optimization skills
• Experience with AWS and or Azure cloud platforms is a plus
• Experience in healthcare data or analytics domain is preferred
• Strong communication, collaboration, and leadership skills
• Ability to lead by example in design, coding, and problem solving

Location and Work Model

• Onsite role based in Bethesda, Maryland
• Open to candidates from DC, Maryland, Baltimore, Virginia, or anywhere in the US
• Candidates must be open to relocation to Bethesda

The HiLabs Story

HiLabs was born in the halls of Yale University when a cardiologist and an AI expert teamed up to tackle data quality challenges in the US healthcare system. Over the years, HiLabs has built one of the most advanced healthcare data platforms, combining AI, data science, and deep domain expertise to deliver real-world impact across the US healthcare ecosystem.

HiLabs Team

• Multidisciplinary industry leaders
• AI, ML, and data science specialists
• Professionals from the world’s top universities and institutes including Harvard, Yale, Carnegie Mellon, Duke, Georgia Tech, Indian Institute of Management, and Indian Institute of Technology.

What We Offer

Competitive base salary, attractive incentive policies, comprehensive benefits including medical coverage for you and your family, 401k, PTOs, employee stock options, relocation support, and an autonomous collaborative working environment. You will work alongside highly qualified professionals from top medical schools, business schools, and engineering institutes, with strong mentorship and long-term growth.