Data Engineer

Speridian; Austin (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Speridian in Austin, TX seeks a seasoned Data Engineer to design, build, and optimize Azure-based data pipelines. You will leverage Azure Data Factory and Databricks to deliver scalable data solutions, integrate data from multiple sources, and support analytics-driven decision making.

The role requires 8–12 years of data engineering experience, strong SQL/NoSQL skills, and expertise in data warehousing and ETL/ELT processes.

Qualifications

• 8–12 years of experience as a data engineer.
• Proficient in designing, developing, and maintaining data pipelines in Azure.
• Expertise with Azure Data Factory (ADF) for data integration and orchestration.
• Experience with Azure Databricks for data processing and analytics.
• Deep knowledge of SQL and NoSQL databases.
• Experience with data warehousing and data modeling.
• Familiarity with ETL/ELT processes and tools.
• Ability to optimize and troubleshoot data pipelines for performance and scalability.
• Experience with version control and deployment tools like Git and Azure DevOps.
• Strong understanding of cloud computing concepts, data security, and privacy.

Responsibilities

• Develop and maintain data pipelines and ETL processes using Azure Data Factory (ADF) and Databricks.
• Design and implement data storage solutions on Azure, including Azure SQL Database, Azure Data Lake Storage, and Azure Blob Storage.
• Create and manage data models and data warehouses using Azure Synapse Analytics.
• Collaborate with cross-functional teams to gather and analyze business requirements and translate them into technical solutions.
• Troubleshoot and optimize data processes to ensure high performance and data quality.
• Implement data security measures to protect sensitive information.
• Stay updated with the latest data engineering technologies and best practices.
• Lead and mentor junior data engineers in the team.
• Provide technical guidance and support to other teams and stakeholders.
• Participate in code reviews and ensure adherence to coding standards and best practices.
• Collaborate with DevOps teams to deploy and manage data pipelines and processes.
• Conduct data analysis and provide insights to support business decisions.
• Develop and maintain documentation for data processes and solutions.
• Identify and troubleshoot data-related issues and provide timely resolutions.

Skills

Azure Data Factory

Azure Databricks

SQL

NoSQL

Data warehousing

ETL/ELT

Git/Azure DevOps

Cloud architecture

Mentoring

Agile

Education

Bachelor's or Master's in Computer Science/Engineering

Tools

Azure Synapse Analytics

Azure Data Lake Storage

Azure Blob Storage

Azure SQL Database

Databricks

Job description

The candidate for this data engineer position brings with them 8 to 12 years of extensive experience in the field, making them a highly skilled and knowledgeable professional. They have a strong expertise in Azure Data Engineering, having worked with the platform for several years and gaining a deep understanding of its capabilities and functionalities. Their proficiency in Azure Data Engineering is complemented by their additional skills in ADF (Azure Data Factory) and Databricks, making them a well-rounded data engineer with a diverse set of technical skills. They have a proven track record of designing, building, and maintaining complex data pipelines and systems using these technologies, and have a thorough understanding of data warehousing, data modeling, and ETL processes. With their extensive experience and expertise, the candidate is well-equipped to handle any data engineering challenges and deliver high-quality solutions to drive business success.

Responsibilities

• Develop and maintain data pipelines and ETL processes using Azure Data Factory (ADF) and Databricks
• Design and implement data storage solutions on Azure, including Azure SQL Database, Azure Data Lake Storage, and Azure Blob Storage
• Create and manage data models and data warehouses using Azure Synapse Analytics
• Collaborate with cross-functional teams to gather and analyze business requirements and translate them into technical solutions
• Troubleshoot and optimize data processes to ensure high performance and data quality
• Implement data security measures to protect sensitive information
• Stay updated with the latest data engineering technologies and best practices
• Lead and mentor junior data engineers in the team
• Provide technical guidance and support to other teams and stakeholders
• Participate in code reviews and ensure adherence to coding standards and best practices
• Collaborate with DevOps teams to deploy and manage data pipelines and processes
• Conduct data analysis and provide insights to support business decisions
• Develop and maintain documentation for data processes and solutions
• Identify and troubleshoot data-related issues and provide timely resolutions

Skills

• 8 to 12 years of experience as a data engineer
• Proficient in designing, developing, and maintaining data pipelines in Azure
• Expertise in using Azure Data Factory (ADF) for data integration and orchestration
• Experience with Azure Databricks for data processing and analytics
• In-depth knowledge of SQL and NoSQL databases
• Experience with data warehousing and data modeling
• Familiarity with ETL/ELT processes and tools
• Ability to optimize and troubleshoot data pipelines for performance and scalability
• Experience with version control and deployment tools such as Git and Azure DevOps
• Strong understanding of cloud computing concepts and architecture
• Experience with data security and privacy best practices
• Excellent problem-solving and analytical skills
• Strong communication and collaboration skills
• Experience working in an Agile environment
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field