OCTO ETL Data Engineer

Collaboredge Inc.; Washington (District of Columbia); On-site; USD 110,000 - 170,000; Full time

Job summary

OCTO Enterprise Data is hiring an ETL Data Engineer to support data curation, processing, and transformation tasks. You will build end-to-end pipelines across diverse data sources, leveraging Azure Data Factory, Synapse, Databricks, and other tools to load and transform data for our Cloud Data Exchange.

You will collaborate with multiple agencies, ensure data quality and security, and drive improvements in data workflows.

Qualifications

• Strong ETL development experience with end-to-end pipelines across multiple sources.
• Ability to gather and document requirements for ETL processes.
• Understanding of data warehousing, data lake, BI, and information management concepts.
• Advise on tools and systems for complex data processing challenges.
• SQL proficiency for relational databases.
• Experience with Oracle, SQL Server, Postgres, DB2, CSV/XML and Parquet/AVRO formats.
• Contribute to data governance to ensure accuracy, consistency, security, and reliability.
• Experience with Azure Data Factory, Synapse, SQL Database, and Data Lake Gen2.
• Experience with Informatica PowerCenter, NiFi, Airflow and FME.
• Visualization/Reporting tools like MicroStrategy, Tableau, Esri ArcGIS (highly desired).
• Strong written and verbal communication skills.
• Customer service excellence for external partners.
• Ability to work independently or in teams.
• Databricks experience highly desirable.

Responsibilities

• Coordinate and deliver complex enterprise data projects.
• Collaborate with stakeholders to define system requirements and constraints.
• Develop and enhance ETL pipelines using cloud solutions.
• Establish cloud/on-prem connectivity across ADLS, ADF, Synapse, Databricks.
• Create end-to-end data pipelines from multiple sources to landing layers or SQL tables.
• Adhere to data governance standards for accuracy and security.
• Automate, monitor, and alert data pipelines and workflows.
• Develop and evolve new systems and enhancements while following standards.
• Provide technical design and coding for large-scale systems.
• Interact with executive and technical users to resolve issues.
• Demonstrate SME-level expertise across environments and cross-platforms.
• Deliver support for applications integrating statewide systems.

Skills

ETL development

End-to-end pipelines

Requirement gathering

Data warehousing concepts

SQL proficiency

Azure data services

Informatica/Airflow/NiFi

Data formats (CSV/XML/Parquet/AVRO)

Data governance

Azure Data Factory/Synapse

Databricks

Python

Stakeholder communication

Education

Bachelor’s degree in Information Technology or related field

Tools

Azure Data Factory

Azure Synapse

SQL Database

Data Lake Storage Gen2

Blob Storage

Databricks

Informatica

Power BI or equivalent visualization tools

Job description

Short Description:

The Enterprise Data team at OCTO requires an ETL data engineer to support data operations for its Cloud Data Exchange. The resource will utilize native Azure tools to perform ETL, data loading and data transformation tasks.

Complete Description:

The ETL data engineer will support the OCTO Enterprise Data team for data curation, processing, and transformation tasks. Specifically, the ETL data engineer will be responsible for the following tasks:

Responsibilities:

• 1. Analyzes, designs, develops, implements, replicates, and supports complex enterprise data projects.
• 2. Interfaces with other agencies, consult with and inform user departments on system requirements, advise on environment constraints and operating difficulties for the current state and advise and resolve problems using cloud solutions and develop and replicate future enhancements to District’s data systems.
• 3. Strong knowledge of Extraction, Transformation and Loading (ETL) processes using frameworks like Azure Data Factory, Synapse, Databricks, Informatica by gathering requirements from the stake holders or analyzing the existing code and perform enhancements or new development.
• 4. Establishing the cloud and on-premise connectivity in different systems like ADLS, ADF, Synapse, Databricks
• 5. Hands on experience in Azure cloud services like Azure Data Factory or Azure Synapse, MSSQL Db, Azure SQL DB, Azure Data Lake Storage Gen2, Blob Storage, Python etc.
• 6. Worked on creating end to end pipelines to load data by reading it from multiple sources or source systems and load to landing layer or SQL tables.
• 7. Familiarity/experience with data integration and data pipeline tools (e.g., Informatica, Synapse, Apache NiFi, Apache Airflow)
• 8. Familiarity/experience with various data formats including database specific (Oracle, SQL Server, DB2, Quickbase), text formats(CSV, XML) and Binary(Parquet, AVRO)
• 9. Develops, standardizes and optimizes existing data workflow/pipelines adhering to best practices.
• 10. Adhere and contribute to enterprise data governance standards by ensuring data accuracy, consistency, security, and reliability.
• 11. Automates, monitors, alerts, and manages data pipelines and workflows.
• 12. Analyzes and evaluates system changes to determine feasibility, provides alternative solutions, back-up, and rollback procedures.
• 13. Works on the development of new systems, upgrades and enhancements to existing systems and ensure systems follow approved standards and consistency after the changes.
• 14. Develops complex programs and reports in database query language.
• 15. Familiarity/experience with data visualization tools.
• 16. Familiarity/experience handling and securing sensitive data based on the level of the sensitivity

Responsibilities:

• 1. Demonstrates expertise in conveying technical and functional concepts for a specific technical specialty.
• 2. Identifies improvements to project standards to achieve high quality services/ products. This is a professional position which may require subject matter expertise consistent with demanding and rare technological skills.
• 3. May require coordination of programming activities being conducted by the application development team
• 4. Confers with other business and technical personnel to resolve problems of intent, inaccuracy, or feasibility of computer processing and project design.
• 5. Works with necessary personnel to determine if modifications are necessary with interested personnel to determine necessity for modifications or enhancements.
• 6. Leverages excellent written and verbal communication skills to develop new business process and programming solutions as directed by business and technical stakeholders.
• 7. May coordinate activities of application developers.11-
• 8. Able to identify best practices and standards for the use of the product.
• 9. Proven track record of hands-on technical design and code work within large complex systems.
• 10. Proven hands-on technical work with a variety of technologies.
• 11. Demonstrated technical expertise integrating a variety of diverse technical environments and cross-platform technologies.
• 12. Delivers support and design for industry specific applications that require integration with statewide systems or applications.
• 13. Interacts with executive level business users or technical experts.
• 14. Advanced experience in the required technical subject matter.
• 15. May function as a niche technical SME (Subject Matter Expert).
• 16. Has proven experience across large and complex implementations and systems.

Minimum Education/Certification Requirements :

Bachelor’s degree in Information Technology or related field or equivalent experience

Candidate Skills Matrix:

Following sections are to be filled by the candidate:

Skills | Required/Desired | No. of Years | How many years of experience candidate has?

• Strong knowledge for development of Extract-Transform-Load (ETL) processes, including end-to-end pipelines with data loading from multiple sources | Required | 15 |
• Ability to gather and document requirements for data extraction, transformation and load processes | Required | 15 |
• Understanding of data warehousing, data lake, business intelligence and information management concepts and standards. | Required | 15 |
• Ability to advise internal and external customers on appropriate tools and systems for complex data processing challenges. | Required | 15 |
• Knowledge and use of SQL for relational databases | Required | 11 |
• Experience with various data formats including database specific (Oracle, SQL, Postgres, DB2), text formats (CSV, XML) and Binary (Parquet, AVRO) | Required | 11 |
• Contribute to enterprise data governance standards by ensuring accuracy, consistency, security and reliability | Required | 7 |
• Strong experience with Microsoft Azure tools such as Azure Data Factory, Synapse, SQL Database, Data Lake Storage Gen2, Blob Storage | Required | 5 |
• Experience with data integration and data pipeline tools such as Informatica PowerCenter, Apache NiFi, Apache Airflow and FME | Required | 5 |
• Experience with visualization and reporting software such as MicroStrategy, Tableau, and Esri ArcGIS | Highly desired | 3 |
• Strong communication skills, both oral and written | Required | 3 |
• Ability to provide excellent customer service to external partners | Required | 3 |
• Ability to work independently or as part of a larger team | Required | 3 |
• Experience performing data functions with Databricks | Highly desired | 3 |

Background check:

• Extensive criminal history background check will be required. We cannot submit candidates with recent histories (go back seven years) of extensive driving, drug, robbery or any other illegal activity. Any criminal activity on the background check will eliminate the candidate from consideration. If selected, please make certain that you inform all candidates that they will have to complete this criminal background check prior to starting. NATIONAL background checks are required; Federal background checks are NOT compliant under this contract. A national background check is a national criminal background check that pulls criminal records from State and County Courts in almost every US State.