Data Engineer - Enterprise Data Platform & Analytics

New York State Technology Enterprise Corporation (NYSTEC); City of Albany (NY); On-site; USD 97,172 - 126,323; Full time

Job summary

NYSTEC is seeking a Data Engineer to design, build, and maintain the enterprise data platform and integration solutions. You will create scalable pipelines, warehouses, lakehouses, and data services to support reporting, analytics, AI initiatives, and data-driven decision making.

You will collaborate with BI and technology teams, implement governance, and ensure data quality, security, and reliability across multiple systems while delivering robust documentation and standards.

Qualifications

• Experience developing enterprise data solutions using Microsoft Fabric.
• Advanced SQL development skills, including query optimization, database design, and performance tuning.
• Strong proficiency in Python and/or PySpark for data engineering, automation, and data transformation.
• Experience designing and developing ETL/ELT solutions and enterprise data integration processes.
• Experience designing and supporting relational databases, data warehouses, lakehouses, and enterprise analytical data models.
• Experience integrating data from enterprise applications, APIs, relational databases, flat files, and cloud services.
• Strong understanding of enterprise data architecture, data modeling, data governance, metadata management, and security best practices.
• Experience implementing source control, testing, deployment, and operational support practices for enterprise data solutions.
• Excellent analytical, troubleshooting, organizational, and communication skills.

Responsibilities

• Design, develop, and maintain scalable enterprise data pipelines that support data integration, analytics, and BI initiatives.
• Design, implement, and support enterprise data architecture, including data warehouses, lakehouses, and analytic repositories.
• Develop and maintain robust ETL/ELT processes to ingest, transform, validate, and deliver data from multiple systems.
• Design and implement logical and physical data models for scalability and maintainability.
• Develop, test, deploy, and maintain data engineering solutions using SQL, Python, and PySpark.
• Integrate data from enterprise applications, APIs, relational databases, flat files, and cloud services with integrity.
• Implement and maintain data governance, metadata management, lineage, security, and access controls.
• Monitor, troubleshoot, and optimize pipelines, databases, and platform performance for reliability and efficiency.
• Collaborate with stakeholders to translate requirements into scalable data solutions.
• Develop technical documentation including solution architecture and procedures.
• Evaluate emerging technologies and recommend improvements to enhance data maturity.
• Partner with BI teams to support reporting, AI, and self-service data initiatives.

Skills

SQL development

Python

PySpark

Data integration

ETL/ELT design

Data governance

Communication

Microsoft Fabric

Education

Bachelor's degree in computer science, information systems, data engineering, information technology or a related discipline

Tools

Microsoft Fabric

Azure Data Factory

Azure Data Lake Storage

Azure SQL

SSIS

SQL Server

Job description

NYSTEC is seeking a Data Engineer to design, build, and maintain the enterprise data platform and integration solutions. You will create scalable pipelines, warehouses, lakehouses, and data services to support reporting, analytics, AI initiatives, and data-driven decision making.

You will collaborate with BI and technology teams, implement governance, and ensure data quality, security, and reliability across multiple systems while delivering robust documentation and standards.