Lead Data Engineer

Bitwise; Richmond (VA); On-site; USD 140,000 - 200,000; Full time

Job summary

Bitwise in United States, Virginia, Richmond seeks a Lead Data Engineer to architect and drive data pipelines on Databricks and ADF. You will lead multi-functional teams, design scalable ETL solutions, and ensure data quality and governance across cloud storages and sources.

The role emphasizes hands-on Databricks development, data modeling, and collaboration with customers to deliver technical solutions and innovations. Strong SQL, Agile, and leadership are essential.

Qualifications

• 8-10 years of experience as a Lead Data Engineer with a strong focus on Databricks & Azure Data Factory.
• Strong experience leading the Data Engineering teams working on ADF & Databricks. Work with the teams providing technical approaches and solutions.
• Proven experience with data warehousing concepts and best practices.
• Strong SQL skills, ability to perform effective querying involving multiple tables and subqueries.
• Strong understanding of data modeling and data quality principles.
• Strong communication & collaboration skills, should be able to work closely with customer driving technical conversations.
• Experience with Agile development methodologies.
• Ability to work independently and as part of a team.
• Lead team and drive through technical challenges and solutions. Drive technical calls with customer
• Focus on continuous value additions and innovations
• Experience with any other ETL tool (Talend, Informatica or similar)
• Good to have Insurance domain knowledge
• Good to have exposure to Gen AI concepts

Responsibilities

• Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.
• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
• Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.
• Work with Delta Lake and other advanced features.
• Leverage Unity Catalog for data governance, access control, and data discovery.
• Experience working with Parquet files & Delta files for data storage and processing.
• Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.
• Perform data quality checks and validation to ensure data accuracy and integrity.

Skills

Lead Data Engineer

Databricks

Azure Data Factory

SQL

Data modeling

Data quality

Agile

Communication

Team leadership

Gen AI concepts

Tools

Databricks

Azure Data Factory

Delta Lake

Unity Catalog

Parquet/Delta files

Azure Blob Storage

ADLS

Synapse

Talend

Informatica

Job description

• 8-10 years of experience as a Lead Data Engineer with a strong focus on Databricks & Azure Data Factory.
• Strong experience leading the Data Engineering teams working on ADF & Databricks. Work with the teams providing technical approaches and solutions.
• Proven experience with data warehousing concepts and best practices.
• Strong SQL skills, ability to perform effective querying involving multiple tables and subqueries.
• Strong understanding of data modeling and data quality principles.
• Strong communication & collaboration skills, should be able to work closely with customer driving technical conversations.
• Experience with Agile development methodologies.
• Ability to work independently and as part of a team.
• Lead team and drive through technical challenges and solutions. Drive technical calls with customer
• Focus on continuous value additions and innovations
• Experience with any other ETL tool (Talend, Informatica or similar)
• Good to have Insurance domain knowledge
• Good to have exposure to Gen AI concepts

Databricks

• Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.
• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
• Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.
• Work with Delta Lake and other advanced features.
• Leverage Unity Catalog for data governance, access control, and data discovery.
• Experience working with Parquet files & Delta files for data storage and processing.
• Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.
• Perform data quality checks and validation to ensure data accuracy and integrity.