AWS Big Data Lead Software Engineer

Aumni; Kentucky; On-site; USD 100,000 - 130,000; Full time

Benefits offered by this job

Comprehensive health care coverage

Retirement savings plan

Tuition reimbursement

Mental health support

Financial coaching

Job summary

TwinThread, based in Kentucky, seeks an AWS Big Data Engineer to design and build applications using leading-edge technologies. The role demands expertise in big data processing using tools like Apache Spark and AWS infrastructure. Ideal candidates should have a minimum of 5 years in software engineering and experience with automation methods and CI/CD principles. Competitive rewards include comprehensive healthcare and various employee benefits aimed at fostering diversity and inclusion.

Qualifications

• 5+ years of applied experience in software engineering.
• Hands-on experience with frameworks and system design.
• Proficiency in automation and continuous delivery.

Responsibilities

• Design and build new applications utilizing leading edge technologies.
• Implement software components for reliability and performance.
• Collaborate with teams to meet sprint objectives.

Skills

Apache Spark

AWS Big Data services

Apache Flink

Distributed datastores

CI/CD methodologies

Automation methods

Education

Formal training or certification in software engineering

Tools

Terraform

Airflow

Job description

AWS Big Data Engineer

Job responsibilities

• Design & build new applications utilizing leading edge technologies and modernize existing applications
• Implement batch & real-time software components consistent with architectural best-practices of reliability, security, operational efficiency, cost-effectiveness and performance
• Ensure quality of deployed code via automated unit, integration & acceptance testing
• Collaborate with multi-national agile development, support and business teams to meet sprint objectives
• Participate in all agile meetings & rituals, including daily standups, sprint planning, backlog reviews, demos, and retrospectives
• Provide level 2 support for production systems
• Learn and apply system processes, methodologies, and skills for the development of secure, stable code and systems
• Hands on application development leveraging distributed compute such as Apache Flink or Spark on very large datasets
• Design and development of applications that leverage the AWS infrastructure, deploying software components on AWS using common compute and storage services such as EC2, EKS, Lambda, S3
• Lead and deliver projects from concept to production across PNI (Personalization and Insights) platform

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years applied experience
• Hands‑on practical experience in frameworks, system design, application development, testing, and operational stability
• Experience with Apache Spark, Apache Flink or similar large‑scale data processing engines
• Experience with distributed datastores (e.g., Cassandra, Red Shift)
• Experience designing, developing and deploying software components on AWS using common compute and storage services such as EC2, EKS, Lambda, S3
• Experience with big data / distributed / cloud technology (AWS Big data services like Lambda, Glue, Glue EMR, performance tuning, streaming, Kafka, entitlements, etc.)
• Experience with Apache Spark, Ray or similar large‑scale data processing engines
• Proficiency in automation and continuous delivery methods
• Proficient in all aspects of the software development life cycle
• Advanced understanding of agile methodologies such as CI/CD, application resiliency, and security
• Demonstrated knowledge of software applications and technical processes within a technical discipline (e.g., cloud, big data, artificial intelligence, machine learning, mobile, etc.)

Preferred qualifications, capabilities, and skills

• Experience building ETL/feature processing pipelines
• Experience using workflow orchestration tools—Airflow, Kubeflow, etc.
• Experience using Terraform to deploy infrastructure‑as‑code to public cloud
• Experience with Linux scripting such as Bash, KSH, or Python
• Certified AWS Cloud Practitioner, Developer or Solutions Architect strongly preferred

We offer a competitive total rewards package including base salary determined based on the role, experience, skill set and location. Eligible roles may receive commission‑based pay and/or discretionary incentive compensation, paid in the form of cash and/or forfeitable equity, awarded in recognition of individual achievements and contributions. We also offer a range of benefits and programs to meet employee needs, based on eligibility. These benefits include comprehensive health care coverage, on‑site health and wellness centers, a retirement savings plan, backup childcare, tuition reimbursement, mental health support, financial coaching and more.

We recognize that our people are our strength and the diverse talents they bring to our global workforce are directly linked to our success. We are an equal‑opportunity employer and place a high value on diversity and inclusion at our company. We do not discriminate on the basis of any protected attribute, including race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy or disability, or any other basis protected under applicable law. We also make reasonable accommodations for applicants’ and employees’ religious practices and beliefs, as well as mental health or physical disability needs.

Equal Opportunity Employer. Disability/Veterans.