Senior Data Engineer

TechDigital Group; Minneapolis (MN); On-site; USD 100,000 - 130,000; Full time

Job summary

TechDigital Group is seeking a highly skilled Senior Data Engineer to develop scalable data solutions. The role involves coaching junior engineers on Azure Databricks, optimizing workflows, and collaborating across teams. Candidates should have over 7 years of experience, particularly in Azure technologies and program languages like Python and SQL.

The ideal candidate will be proactive in problem-solving, possess strong communication skills, and be able to work with cross-functional teams effectively.

Qualifications

• 7+ years of experience in data engineering or related roles.
• 5+ years of experience in Azure Databricks, Azure Data Lake, and Azure Cosmos DB.
• Hands-on experience with cloud platforms, particularly Microsoft Azure.

Responsibilities

• Develop and implement scalable data pipelines using Azure Databricks and Azure Data Lake.
• Create and maintain ETL workflows to ensure data quality.
• Monitor and troubleshoot data pipelines and resolve issues promptly.

Skills

Azure Databricks

Azure Data Lake

Azure Cosmos DB

Python

SQL

Scala

Apache Spark

Hadoop

Hive

CI/CD pipelines

DevOps tools

Education

Bachelor's or Master's degree in Computer Science or IT

Tools

Azure Data Factory

Azure Synapse Analytics

Apache Kafka

Docker

Kubernetes

Job description

Job Summary

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing and delivering scalable data solutions using Azure Databricks, Azure Data Lake, and Azure Cosmos DB. This role will involve coaching data engineers technically on topics like Azure Databricks, optimizing data workflows, and collaborating with internal teams on various aspects.

Responsibilities

1. Develop and implement scalable data pipelines using Azure Databricks and Azure Data Lake.
2. Create and maintain ETL workflows to ensure data quality, integrity, and availability.
3. Optimize Azure Cosmos DB schemas and queries for high performance and cost-efficiency.
4. Manage and analyze data at scale within the Azure ecosystem.
5. Coach junior data engineers on technologies such as Azure Databricks and Azure Cosmos DB.
6. Implement CI/CD pipelines using Azure DevOps to streamline deployment processes.
7. Conduct code reviews, provide mentorship, and facilitate knowledge sharing sessions within the team.
8. Integrate data from various sources into a centralized data lake, enabling unified access and analysis.
9. Perform data cleansing, transformation, and migration tasks as required.
10. Monitor and troubleshoot data pipelines and resolve any issues promptly.

Additional Responsibilities

1. Provide technical guidance to a team of developers, enhancing their technical capabilities and increasing productivity.
2. Conduct comprehensive code reviews, establish and oversee quality assurance processes, performance optimization, and implementation of best practices and coding standards to ensure successful delivery of complex projects.
3. Ensure process compliance in the assigned module and participate in technical discussions or reviews as a technical consultant for feasibility study, technical alternatives, best packages, supporting architecture best practices, technical risks, breakdown into components, estimations.
4. Collaborate with stakeholders to define project scope, objectives, deliverables and accordingly prepare and submit status reports for minimizing exposure and closure of escalations.

Requirements

1. Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
2. 7+ years of experience in data engineering or related roles.
3. 5+ years of experience in Azure Databricks, Azure Data Lake, and Azure Cosmos DB with a proven track record of building data engineering solutions.
4. Proficient in programming languages such as Python, SQL, and Scala.
5. Experience with big data technologies including Apache Spark, Hadoop, and Hive.
6. Hands-on experience with cloud platforms, particularly Microsoft Azure.
7. Proficiency in ETL processes, data warehousing, and data modeling.
8. Proficient in using Azure Cosmos DB for graph-based data analysis.
9. Knowledge of tools and technologies such as Azure Data Factory, Azure Synapse Analytics, and Apache Kafka.
10. Solid understanding of CI/CD pipelines and hands-on experience with DevOps tools such as Git for continuous integration and deployment.
11. Excellent problem-solving skills and the ability to troubleshoot complex data issues.
12. Strong coaching abilities, excellent communication and collaboration skills, and the ability to work effectively with cross-functional teams.

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate.
• Databricks Certified Associate Developer for Apache Spark.
• Experience with data governance and data security practices.
• Knowledge of Docker and Kubernetes for containerization and orchestration.