Data Engineer | Python Development

PradeepIT Consulting Services Pvt Ltd; Hyderabad; Hybrid; INR 800,000 - 1,200,000; Full time

Job summary

A leading consulting firm is seeking a Data Engineer to develop and manage data pipelines using AWS technologies. The candidate must have 3-5 years of experience with Python development and AWS services like Lambda and S3. The role supports a hybrid work environment, requiring collaboration with business analysts and robust problem-solving skills.

Qualifications

• 3-5 years of Python development experience including libraries for data processing.
• Strong knowledge of AWS services and serverless architecture.
• Experience building data pipelines and workflows.

Responsibilities

• Develop, test, and deploy data processing pipelines using AWS technologies.
• Implement ETL processes for structured and unstructured data.
• Collaborate with teams to understand requirements and deliver solutions.

Skills

Python development

AWS services

Data processing (Pandas, boto3)

Version control (Git)

CI/CD practices

Tools

AWS Lambda

DynamoDB

S3

Step Functions

Job description

About the job Data Engineer | Python Development

Data Engineer with Python development experience

Experience: 3+ Years

Mode: Hybrid (2-3 days/week)

Key Responsibilities

• Develop, test, and deploy data processing pipelines using AWS Serverless technologies such as AWS Lambda, Step Functions, DynamoDB, and S3.
• Implement ETL processes to transform and process structured and unstructured data efficiently.
• Collaborate with business analysts and other developers to understand requirements and deliver solutions that meet business needs.
• Write clean, maintainable, and well-documented code following best practices.
• Monitor and optimize the performance and cost of serverless applications.
• Ensure high availability and reliability of the pipeline through proper design and error handling mechanisms.
• Troubleshoot and debug issues in serverless applications and data workflows.
• Stay up-to-date with emerging technologies in the AWS and serverless ecosystem to recommend improvements.

Required Skills and Experience

• 3-5 years of hands-on Python development experience, including experience with libraries like boto3, Pandas, or similar tools for data processing.
• Strong knowledge of AWS services, especially Lambda, S3, DynamoDB, Step Functions, SNS, SQS, and API Gateway.
• Experience building data pipelines or workflows to process and transform large datasets.
• Familiarity with serverless architecture and event-driven programming.
• Knowledge of best practices for designing secure and scalable serverless applications.
• Proficiency in version control systems (e.g., Git) and collaboration tools.
• Understanding of CI/CD pipelines and DevOps practices. Strong debugging and problem-solving skills.
• Familiarity with database systems, both SQL (e.g., RDS) and NoSQL (e.g., DynamoDB).

Preferred Qualifications

• Familiarity with testing frameworks (e.g., pytest) and ensuring test coverage for Python applications.
• Experience with Infrastructure as Code (IaC) tools such as AWS CDK, CloudFormation.
• Knowledge of monitoring and logging tools.