We are looking for an experienced AWS Big Data Engineer with strong expertise in Python/Scala, Apache Spark, AWS cloud services, and Apache Airflow. The ideal candidate will have hands‑on experience developing scalable data engineering solutions, API‑driven backend services, and distributed Big Data applications.
Key Responsibilities
- Design, develop, test, and deploy scalable Big Data applications and data engineering solutions on AWS.
- Develop API-driven backend services to manage, monitor, and govern large‑scale batch data processing applications.
- Build and maintain data pipelines using Apache Spark, Python, Scala, SQL, Hive, and Hadoop.
- Work with AWS services including S3, EC2, EMR, Lambda, DynamoDB, and API Gateway.
- Design and enhance Apache Airflow DAGs for workflow orchestration, scheduling, dependency management, monitoring, and failure handling.
- Develop scalable batch and real‑time data processing solutions.
- Perform performance tuning, optimization, troubleshooting, and production support for distributed data applications.
- Participate in architecture reviews, code reviews, testing, and deployment activities.
- Collaborate with data engineers, backend developers, QA, DevOps, and product teams in an Agile/Scrum environment.
- Implement automation, CI/CD practices, and operational improvements to enhance system reliability.
Mandatory Skills
- 5+ years of experience in Big Data Engineering, distributed systems, or enterprise application development.
- Strong hands‑on experience with AWS cloud services, particularly S3, EC2, EMR, Lambda, DynamoDB, and API Gateway.
- Strong programming experience in Python or Scala.
- Hands‑on experience with Apache Spark, Hadoop, and Hive.
- Strong experience in Apache Airflow, including complex DAG development, scheduling, dependencies, monitoring, and error handling.
- Strong SQL and data engineering fundamentals.
- Experience developing APIs, backend services, or API‑driven applications.
- Experience with Git, Jenkins, and CI/CD pipelines.
- Good understanding of batch processing, data transformation, data quality, and performance optimization.
- Experience working in Agile/Scrum environments.
Good to Have
- Exposure to LLMs, Generative AI, Agentic AI, or AI‑assisted engineering.
- Experience with REST APIs, microservices, event‑driven architecture, and serverless applications.
- Knowledge of Infrastructure as Code and automated testing.li>
- Experience with data governance, metadata management, data lineage, and data observability.
- Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical discipline.
Key Skills
- AWS
- AWS EMR
- Apache Spark
- Python
- Scala
- Apache Airflow
- Hadoop
- Hive
- SQL
- S3
- EC2
- Lambda
- DynamoDB
- API Gateway
- REST APIs
- Microservices
- Git
- Jenkins
- CI/CD
Skills
- aws,rest apis,python,ci/cd,sql,microservices,git,scala,big data