Azure Data Engineer

Q1 Technologies, Inc

Las Vegas (NV)

On-site

USD 120,000 - 160,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Q1 Technologies, Inc. is seeking an Azure Data Engineer to design, build, and maintain data pipelines and ELT processes.

You will implement custom connectors for diverse data sources, use Apache Airflow and PySpark for data orchestration, and collaborate with cross-functional teams to deliver robust data solutions in a fast-paced environment. You will apply DataOps and CI/CD practices, monitor pipelines, validate data, and keep up-to-date with evolving technologies while documenting workflows

Qualifications

  • Bachelor's degree in computer science, engineering, or a related field.
  • 8 years of experience in data engineering, ELT development, and data modeling.
  • Proficiency in Apache Airflow and Spark for data transformation, integration, and management.
  • Experience implementing workflow orchestration using Airflow, SSIS or similar platforms.
  • Demonstrated experience developing custom connectors for data ingestion from various sources.
  • Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
  • Experience implementing DataOps principles and practices, including data CI/CD pipelines.
  • Excellent problem-solving and troubleshooting skills with attention to detail.
  • Effective communication and collaboration abilities in cross-functional teams.
  • Familiarity with Apache Superset and dashboard development.
  • Understanding of distributed systems and large-scale datasets.
  • Familiarity with data governance frameworks and practices.
  • Knowledge of data streaming technologies (e.g., Apache Kafka).
  • Strong understanding of software development principles, including Git and code reviews.
  • Experience with Agile methodologies in cross-functional teams.

Responsibilities

  • Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark.
  • Design and implement custom connectors to ingest data from diverse sources into the platform.
  • Collaborate with cross-functional teams to gather requirements and translate them into technical solutions.
  • Implement DataOps principles and CI/CD pipelines for robust data operations.
  • Monitor, troubleshoot, and optimize data ingestion, transformation, and loading processes.
  • Perform data validation and testing to ensure accuracy and governance.
  • Stay current with emerging data engineering technologies and best practices.
  • Document data workflows and technical specifications for knowledge sharing.

Skills

Python
PySpark
SQL
Data modeling
Airflow
DataOps
Agile
Problem solving
Communication

Education

Bachelor's degree in CS/Engineering or related field

Tools

SSIS
Apache Superset
Kafka
Git

Job description

Job Description

Job Description Role: Azure Data Engineer

Descriptions:

Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data.

Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .

Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.

Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery.

Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.

Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.

Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.

Stay up-to-date with emerging technologies and best practices in data engineering.

Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

Experience:

Bachelor's degree in computer science, Engineering, or a related field

8 year's experience in data engineering, ELT development, and data modeling.

Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.

Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.

Demonstrated experience in developing custom connectors for data ingestion from various sources.

Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.

Experience implementing DataOps principles and practices, including data CI/CD pipelines.

Excellent problem-solving and troubleshooting skills, with a strong attention to detail.

Effective communication and collaboration abilities, with a proven track record of working in cross-functional teams.

Familiarity with data visualization tools Apache Superset and dashboard development.

Understanding of distributed systems and working with large-scale datasets.

Familiarity with data governance frameworks and practices.

Knowledge of data streaming and real-time data processing technologies (e.g., Apache Kafka).

Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes.

Experience with Agile development methodologies and working in cross-functional Agile teams.

Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.

Excellent analytical and problem-solving skills, with a keen attention to detail.

Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Azure Data Engineer
Azure Data Engineer

Robotic Process Automation LLC • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Recru • Fort Worth (TX), Arlington (TX), Dallas (TX)

On-site
USD 120,000 - 150,000
Azure Data Engineer
Azure Data Engineer

Robotics Technologies • Raleigh (NC)

On-site
USD 110,000 - 150,000
Azure Data Engineer
Azure Data Engineer

Ethereum Technologies LLC • Raleigh (NC)

On-site
USD 120,000 - 150,000
None
Azure Data Engineer
Azure Data Engineer

Valiant Technologies LLC • Raleigh (NC)

On-site
USD 110,000 - 150,000
Azure Data Engineer
Azure Data Engineer

CloudBigD • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Quantum Technologies LLC • Raleigh (NC)

On-site
USD 110,000 - 160,000
Data Engineer
Data Engineer

EdgeQ Technologies Private Limited • Dallas (TX)

On-site
USD 100,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Veriipro • Bellevue (WA)

On-site
USD 110,000 - 160,000
Data Engineer
Data Engineer

Compunnel Inc. • Norfolk (VA)

On-site
USD 110,000 - 150,000