Azure ETL Data Engineer - Cloud Data Pipelines Lead

Collaboredge Inc.; Washington (District of Columbia); On-site; USD 110,000 - 170,000; Full time

Job summary

OCTO Enterprise Data is hiring an ETL Data Engineer to support data curation, processing, and transformation tasks. You will build end-to-end pipelines across diverse data sources, leveraging Azure Data Factory, Synapse, Databricks, and other tools to load and transform data for our Cloud Data Exchange.

You will collaborate with multiple agencies, ensure data quality and security, and drive improvements in data workflows.

Qualifications

• Strong ETL development experience with end-to-end pipelines across multiple sources.
• Ability to gather and document requirements for ETL processes.
• Understanding of data warehousing, data lake, BI, and information management concepts.
• Advise on tools and systems for complex data processing challenges.
• SQL proficiency for relational databases.
• Experience with Oracle, SQL Server, Postgres, DB2, CSV/XML and Parquet/AVRO formats.
• Contribute to data governance to ensure accuracy, consistency, security, and reliability.
• Experience with Azure Data Factory, Synapse, SQL Database, and Data Lake Gen2.
• Experience with Informatica PowerCenter, NiFi, Airflow and FME.
• Visualization/Reporting tools like MicroStrategy, Tableau, Esri ArcGIS (highly desired).
• Strong written and verbal communication skills.
• Customer service excellence for external partners.
• Ability to work independently or in teams.
• Databricks experience highly desirable.

Responsibilities

• Coordinate and deliver complex enterprise data projects.
• Collaborate with stakeholders to define system requirements and constraints.
• Develop and enhance ETL pipelines using cloud solutions.
• Establish cloud/on-prem connectivity across ADLS, ADF, Synapse, Databricks.
• Create end-to-end data pipelines from multiple sources to landing layers or SQL tables.
• Adhere to data governance standards for accuracy and security.
• Automate, monitor, and alert data pipelines and workflows.
• Develop and evolve new systems and enhancements while following standards.
• Provide technical design and coding for large-scale systems.
• Interact with executive and technical users to resolve issues.
• Demonstrate SME-level expertise across environments and cross-platforms.
• Deliver support for applications integrating statewide systems.

Skills

ETL development

End-to-end pipelines

Requirement gathering

Data warehousing concepts

SQL proficiency

Azure data services

Informatica/Airflow/NiFi

Data formats (CSV/XML/Parquet/AVRO)

Data governance

Azure Data Factory/Synapse

Databricks

Python

Stakeholder communication

Education

Bachelor’s degree in Information Technology or related field

Tools

Azure Data Factory

Azure Synapse

SQL Database

Data Lake Storage Gen2

Blob Storage

Databricks

Informatica

Power BI or equivalent visualization tools

Job description

OCTO Enterprise Data is hiring an ETL Data Engineer to support data curation, processing, and transformation tasks. You will build end-to-end pipelines across diverse data sources, leveraging Azure Data Factory, Synapse, Databricks, and other tools to load and transform data for our Cloud Data Exchange.

You will collaborate with multiple agencies, ensure data quality and security, and drive improvements in data workflows.