Lead Data Engineer

Choosecolumbusga; Columbus (GA); Hybrid; USD 100,000 - 150,000; Full time

Job summary

Choosecolumbusga seeks a Lead Data Engineer to join their Corporate Data Technology Team in Columbus, Ohio. This role involves designing scalable data frameworks, developing robust data pipelines, and collaborating with business teams to meet technical specifications.

The ideal candidate will have at least ten years of experience in IT, strong knowledge of Databricks and Azure services, and the ability to ensure data quality and integrity. This position is vital for driving innovation with data solutions.

Qualifications

• Ten years of experience in information technology involving data usage.
• Experience with automation, data mapping, and conducting quality assessments.
• Strong verbal and written communication skills.

Responsibilities

• Design and implement scalable data frameworks for data analytics.
• Develop secure and high-quality data pipelines using Databricks.
• Implement data quality checks to ensure data reliability.

Skills

Data processing frameworks

Python

Azure data services

Databricks

Data Quality checks

Education

Bachelor's degree in computer science or related field

Tools

Databricks

Tableau

SQL Server

Job description

Job Summary

Works with business partners as the primary interface between Information Technology and specific business areas, identifying business unit requirements, creating project and process specifications, coordinating with project teams, and ensuring adherence to schedules and budgets and alignment with Information Technology strategy. Determines if solutions already exist or if new solutions are needed and provides input regarding buy vs build decisions. Ensures proper integration of solutions that impact work processes. Identifies opportunities for improvement or enhancement and negotiates agreements and commitments between business partners and Information Technology from requirements gathering to final implementation.

Job Duties and Responsibilities

• Serves as an expert regarding the business units supported and has expert understanding of the Information Technology organization's systems and capabilities. Analyzes business unit operations to understand strengths and weaknesses and to identify process and function automation opportunities. Develops appropriate business models to represent business requirements and meet business needs.
• Obtains and validates information requirements from stakeholders through various analysis techniques. Translates high level business requirements into functional specifications for the Information Technology organization. Manages requirements, approvals, changes, and traceability.
• Assists with business process redesign and documentation as needed for new technology. Ensures technology solutions are in compliance with and uphold organizational standards. Develops business models to represent requirements, such as use cases, business process flows, and data flow diagrams.
• Negotiates agreements and commitments. Facilitates communication between business units and Information Technology from initial requirements to final implementation. Provides input regarding buy versus build decisions. Participates in planning sessions with business units to improve business processes.
• Participates in defining and streamlining division level processes, procedures, and templates as needed to support the organization's process maturity goals and to ensure effective and efficient methods in accordance with best practices.
• Provides business reporting and analytical support, particularly pertaining to departmental governance and performance, utilizing various data sources and analytical tools. Performs analysis and provides reporting views with improvement recommendations. Designs, develops, tests, and maintains various reports using SQL driven application table portlets and Business Objects.
• Conducts systems analysis with a focus on business process models and use cases. Performs system administration and maintenance as needed in support of departmental activities. Performs audits to ensure compliance with division processes, procedures and standards.
• Analyzes incidents related to the system/application to identify problem areas. Works closely with the application business partner's product support team to identify and implement solutions.
• Works on multiple projects evaluating and identifying project complexity, assumptions, constraints, and dependencies. Conducts timely surveys to gauge effectiveness of delivery of technology solutions.
• Performs a variety of tasks to improve on the existing operating processes, methodologies and standards. Participates in the development of processes, procedures, and protocols for the department to ensure effective and efficient methods of providing support. Coordinates the maintenance of all tools used to administer the processes as needed to ensure system currency.
• Builds, maintains, and manages stakeholder relationships. Educates the Information Technology organization on the direction of the business. Communicates complex technical concepts simply and effectively to nontechnical team members. Guides team members through the established operating process and escalates non-conformance as needed.
• Maintains a current knowledge base and understanding of technological trends. Uses Information Technology knowledge base to bring solutions to business units supported and enhances the enterprise's competitive edge.
• Assists less experienced team members as needed.
• Each team member is expected to be aware of risk within their functional area. This includes observing all policies, procedures, laws, regulations and risk limits specific to their role. Additionally, they should raise and report known or suspected violations to the appropriate Firm authority in a timely fashion.
• Performs other related duties as required.

Minimum Education

Bachelor's degree in computer science, information systems, or a related field or an equivalent combination of education and experience.

Minimum Experience

10 years of experience developing, implementing and maintaining enterprise policies, procedures and operating standards including preparing agendas, meeting minutes, and follow-up actions for meetings with management. Regular office environment;

Required Knowledge, Skills & Abilities

• May sit for long periods of time;
• Talk and hear regularly;
• Use of hands, ability to reach;
• Ability to see objects up close or at a distance, use peripheral vision, identify basic colors;
• Up to 10 pounds weight lifting/force exertion required;
• Weight/force exertion required less than 25% of the time;
• Travel required less than 25% of the time

Preferred Knowledge, Skills, & Abilities

• 7+ years of experience in data engineering, including design, application development, testing, and operational stability.
• Advanced proficiency in data processing frameworks and tools, including Parquet, Iceberg, PySpark, Lambda, Databricks and Azure data services.
• Proficiency in programming languages such as Python, Java, or Scala for data processing and application development.
• Extensive hands‑on experience with Databricks, including architecting and managing large‑scale data pipelines, optimizing Spark jobs, implementing security and governance controls, and automating workflows. Proven ability to leverage Databricks for advanced analytics, data lakehouse architectures, and seamless integration with AWS or other cloud services.
• Proficiency in automation and continuous delivery methods, utilizing CI/CD pipelines with tools like Git/Bitbucket, Jenkins, automated DAB deployments and version control.
• Certification on Databricks Data Engineering Associate or Professional.
• Azure Certifications.
• Proficiency in relational databases (Oracle or SQL Server and ER Data Modeling.
• Skilled in writing Oracle SQL queries utilizing DML, DDL, and

The information on this description has been designed to indicate the general nature and level of work performed by employees within this classification. It is not designed to contain or be interpreted as a comprehensive inventory of all duties, responsibilities, and qualifications required of employees assigned to this job

Pinnacle is an Equal Opportunity Employer committed to fostering an inclusive work environment.