Data Engineer

Brooksource; Charlotte (NC); On-site; USD 90,000 - 140,000; Full time

Job summary

Brooksource is seeking an experienced data engineer to design, build, and optimize scalable data pipelines using SQL and Python in an enterprise AWS/Snowflake environment. You will help shape ingestion, transformation, and analytics layers and contribute to AI-powered tooling to accelerate delivery.

The role encompasses data modeling, governance, observability, and cost optimization across data products. You will collaborate with agile teams to translate requirements into robust technical

Responsibilities

• Design, develop, and maintain scalable batch and near real-time data pipelines using SQL and Python.
• Build, optimize, and support data ingestion, transformation, and orchestration processes in Snowflake and AWS.
• Develop reusable data assets, curated datasets, and data models that support analytics, reporting, operational workflows, and AI solutions.
• Create and maintain ETL/ELT frameworks to integrate data from multiple source systems.
• Ensure data solutions are scalable, reliable, secure, and cost-effective.
• Leverage AWS services such as S3, Lambda, Glue, ECS, and other cloud-native technologies to enable enterprise data processing and storage.
• Support cloud data warehouse and data lake architectures.
• Monitor, tune, and optimize data workloads to improve performance, reliability, and cost efficiency.
• Understand data quality, governance, lineage, and observability capabilities across data products and platforms.
• Leverage AI powered development tools and coding assistants to improve engineering productivity, accelerate software delivery, and enhance code quality.
• Utilize generative AI capabilities to support code generation, documentation creation, testing, troubleshooting, and data pipeline development.
• Identify opportunities to automate manual engineering processes through AI-enabled workflows and tooling.
• Evaluate and adopt emerging AI technologies and best practices while adhering to enterprise security, governance, and responsible AI standards.
• Utilize GitLab for source control, CI/CD pipelines, automated testing, code reviews, and deployment automation.
• Implement DevOps best practices to improve delivery speed, quality, reliability, and operational support.
• Participate in production support, incident management, root cause analysis, and continuous improvement activities.
• Develop and maintain technical documentation, standards, and reusable engineering components.
• Participate in Agile Scrum ceremonies including sprint planning, backlog refinement, daily standups, sprint reviews, and retrospectives.
• Collaborate with cross-functional teams to translate business requirements into scalable technical solutions.
• Contribute to architecture, data modeling, and design discussions.

Job description

• Design, develop, and maintain scalable batch and near real-time data pipelines using SQL and Python.
• Build, optimize, and support data ingestion, transformation, and orchestration processes in Snowflake and AWS.
• Develop reusable data assets, curated datasets, and data models that support analytics, reporting, operational workflows, and AI solutions.
• Create and maintain ETL/ELT frameworks to integrate data from multiple source systems.
• Ensure data solutions are scalable, reliable, secure, and cost-effective.
• Leverage AWS services such as S3, Lambda, Glue, ECS, and other cloud-native technologies to enable enterprise data processing and storage.
• Support cloud data warehouse and data lake architectures.
• Monitor, tune, and optimize data workloads to improve performance, reliability, and cost efficiency.
• Understand data quality, governance, lineage, and observability capabilities across data products and platforms.

AI Enabled Engineering Productivity

• Leverage AI powered development tools and coding assistants to improve engineering productivity, accelerate software delivery, and enhance code quality.
• Utilize generative AI capabilities to support code generation, documentation creation, testing, troubleshooting, and data pipeline development.
• Identify opportunities to automate manual engineering processes through AI-enabled workflows and tooling.
• Evaluate and adopt emerging AI technologies and best practices while adhering to enterprise security, governance, and responsible AI standards.
• Utilize GitLab for source control, CI/CD pipelines, automated testing, code reviews, and deployment automation.
• Implement DevOps best practices to improve delivery speed, quality, reliability, and operational support.
• Participate in production support, incident management, root cause analysis, and continuous improvement activities.
• Develop and maintain technical documentation, standards, and reusable engineering components.

Agile Delivery & Collaboration

• Participate in Agile Scrum ceremonies including sprint planning, backlog refinement, daily standups, sprint reviews, and retrospectives.
• Collaborate with cross-functional teams to translate business requirements into scalable technical solutions.
• Contribute to architecture, data modeling, and design discussions.