Design and operate enterprise-grade data platforms for the Department of Innovation and Technology (DoIT) within a hybrid work setting in Chicago. As a Senior Data Engineer, you will lead the creation and optimization of large-scale data pipelines and distributed processing systems that support analytics, operations, and regulatory reporting, while strengthening governance, reliability, and visibility across the state’s data estate.
In this role, you will serve as a senior technical contributor and advisory partner across cross-functional data delivery squads, helping establish enterprise data engineering standards. You will also have the opportunity to grow your skills through ongoing education and professional development, including online training, meetings, seminars, and conferences.
Responsibilities
- Under administrative direction, perform highly complex professional and advisory functions for DoIT, including leading the design, development, and optimization of enterprise data pipelines and scalable data platforms.
- Design, build, and optimize enterprise ETL and ELT pipelines, distributed processing workloads, and semantic data models for analytics, business intelligence, and operational reporting.
- Lead technical governance for enterprise data lake architecture, with focus on high availability, data quality, cost-efficient storage tiering, and end-to-end observability across the enterprise data asset inventory.
- Design, implement, and govern CI/CD automation frameworks for data engineering workloads to support code promotion, environment parity, and production deployment that align with enterprise reliability and security standards.
- Implement, govern, and continuously improve role-based access control for data platforms, ensuring data access policy is enforced technically, auditable, and aligned with DoIT security standards and applicable state and federal data governance regulations.
- Engineer and optimize data workloads across relational and non-relational platforms and emerging technologies, including large-scale RDBMS and systems such as document databases, key value stores, and graph databases.
- Provide senior technical leadership in cross-functional squads, representing enterprise data engineering standards in feature development and advising vendor partners and executive stakeholders.
- Maintain current knowledge through continuing education, training sessions, seminars, and conferences.
- Perform other duties as required or assigned within the scope of the position.
Requirements
- Knowledge, skill, and mental development equivalent to four (4) years of college with coursework in Computer Science, Computer Engineering, Information Systems, Data Engineering, Software Engineering, Management Information Science, or a closely related field.
- Five (5) years of professional experience in a related Information Technology field, including data engineering, software engineering, database administration, business intelligence development, data architecture, systems integration, or closely related work.
- Five (5) years designing, building, and optimizing enterprise-scale data pipelines, fault-tolerant ETL/ELT patterns, and distributed data processing systems in a high-volume near-time production environment, including hands‑on SQL performance tuning and execution plan analysis across platforms such as Microsoft SQL Server, Azure SQL Database, DB2, or equivalent.
- Five (5) years with distributed data processing and Spark execution engine optimization, including tuning large-scale batch, micro‑batch, and streaming workloads, with proficiency in execution plans, shuffle and partition strategy, memory management, and cluster autoscaling across platforms such as Databricks, Apache Spark, Azure Synapse, Teradata Vantage, Snowflake, Amazon EMR, Starburst, or Cloudera.
- Five (5) years with enterprise data platform and lakehouse architecture design, including multi‑zone lake and lakehouse patterns, open table format optimization, schema evolution, and integration of storage, compute, and catalog layers across platforms such as Microsoft Fabric, Databricks Unity Catalog, Snowflake, or equivalent.
- Four (4) years engineering reusable data access layers and gateway patterns that abstract source system complexity, reduce redundant data movement, and provide consistent governed interfaces for downstream consumers.
Technologies
- SQL, ETL, ELT, CI/CD, DataOps, Spark
- Microsoft SQL Server, Azure SQL Database, DB2
- MongoDB, Couchbase, Redis, DynamoDB
- Neo4j, Cosmos DB Gremlin API
- Databricks, Apache Spark, Azure Synapse, Teradata Vantage, Snowflake, Amazon EMR, Starburst, Cloudera
- Microsoft Fabric, Databricks Unity Catalog
- Terraform, Azure Bicep, Databricks Asset Bundles, GitHub Actions
Benefits
- Competitive group insurance benefits including health, life, dental, and vision.
- Flexible work schedules when available and dependent upon position.
- 10 to 25 days of paid vacation annually (10 days for first year of state employment).
- 12 days of paid sick time annually, carry over year to year.
- 3 paid personal business days per year.
- 13 to 14 paid holidays per year depending on election years.
- 12 weeks of paid parental leave.
- Pension plan through the State Employees Retirement System.
- Deferred Compensation Program, voluntary supplemental retirement plan.
- Optional pre‑tax programs: Medical Care Assistance Plan (MCAP) and Dependent Care Assistant Plan (DCAP).
- Tuition Reimbursement Program and Federal Public Service Loan Forgiveness Program eligibility.
- For more information on State of Illinois benefits: https://www2.illinois.gov/cms/benefits/Pages/default.aspx
Job Details
- Location: Chicago, IL (hybrid); 1 to 5 days per week in office depending on operational needs
- Salary: Anticipated starting salary $9,130 per month; full range $9,130-$13,573
- Work Hours: 8:30 AM to 5:00 PM, Monday through Friday (hours may vary)
- Job Type: Salaried; Full Time
- Number of Vacancies: 1
- Union position: AFSCME (see collective bargaining agreement/labor contract provisions)
- Agency: Department of Innovation and Technology
- Headquarter Location: 555 W Monroe St, Chicago, Illinois, 60661
- Agency Contact: Lorna.A.Bock@illinois.gov
Note on sponsorship: The State of Illinois does not provide sponsorship for visa status (such as H-1B) and cannot provide OPT extensions under STEM-designated degrees. Applicants must be currently authorized to work in the United States on a full‑time basis.
Reasonable accommodation: The State of Illinois is committed to working with and providing reasonable accommodations for people with disabilities. To request an accommodation during the hiring process, contact CMS.DisabilityResCen@illinois.gov or call (217) 524-7514.