Lead Data Engineer

Bitwise; Richmond (VA); On-site; USD 150,000 - 190,000; Full time

Job summary

Bitwise in Richmond, VA is seeking a Lead Data Engineer with 8-10 years of experience to lead data pipelines using Databricks and Azure Data Factory. You will architect scalable data solutions and mentor a growing engineering team.

You will collaborate with customers, drive technical discussions, ensure data quality and governance with Delta Lake and Unity Catalog, and steer agile delivery across projects.

Qualifications

• 8-10 years of experience as Lead Data Engineer, with focus on Databricks and Azure Data Factory.
• Proven ability to lead data engineering teams, drive technical approaches and customer conversations.
• Strong SQL, data warehousing concepts, data modeling and data quality.
• Experience with Agile development and working independently or in a team.
• Experience with Talend or Informatica is a plus.

Responsibilities

• Lead data engineering teams on Databricks and ADF initiatives.
• Design, develop, and deploy ETL pipelines and notebooks.
• Collaborate with customers to drive technical solutions and architecture.
• Establish data governance and quality controls with Delta Lake and Unity Catalog.
• Mentor teammates and drive continuous value additions and innovations.

Skills

Databricks

Azure Data Factory

SQL

Data modeling

Data quality

Agile

Team leadership

Communication

ETL pipelines

Gen AI concepts

Tools

Databricks

Azure Data Factory

Delta Lake

Unity Catalog

Parquet

Delta files

Azure Blob Storage

ADLS

Synapse

Talend

Informatica

Job description

• 8-10 years of experience as a Lead Data Engineer with a strong focus on Databricks & Azure Data Factory.
• Strong experience leading the Data Engineering teams working on ADF & Databricks. Work with the teams providing technical approaches and solutions.
• Proven experience with data warehousing concepts and best practices.
• Strong SQL skills, ability to perform effective querying involving multiple tables and subqueries.
• Strong understanding of data modeling and data quality principles.
• Strong communication & collaboration skills, should be able to work closely with customer driving technical conversations.
• Experience with Agile development methodologies.
• Ability to work independently and as part of a team.
• Lead team and drive through technical challenges and solutions. Drive technical calls with customer.
• Focus on continuous value additions and innovations.
• Experience with any other ETL tool (Talend, Informatica or similar).
• Good to have Insurance domain knowledge.
• Good to have exposure to Gen AI concepts.

Databricks

• Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.
• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
• Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.
• Work with Delta Lake and other advanced features.
• Leverage Unity Catalog for data governance, access control, and data discovery.
• Experience working with Parquet files & Delta files for data storage and processing.
• Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.
• Perform data quality checks and validation to ensure data accuracy and integrity.