Senior BigData Engineer
NewVision Software
5 days ago
Be an early applicant
Get more replies from employers
Send a job-specific resume in minutes.
Job summary
NewVision Software in Pune, India seeks an experienced Big Data Engineer to design distributed architectures and build scalable batch and streaming data pipelines.
You will work with AWS, Snowflake, Spark, and containerized/serverless technologies, ensuring data quality, governance, and secure data handling while collaborating with cross-functional teams and mentoring junior engineers.
Qualifications
- 5+ years of industrial experience in Big Data Engineering
- Strong proficiency in SQL and at least one programming language: Python, Scala, or Java
- Hands-on experience with AWS and Snowflake
- Strong experience in DBT
- Experience with cloud storages: AWS S3, Azure Storage, Google Cloud Storage
- Practical skills with Hadoop-like environments (EMR, HDInsight, Dataproc)
- Experience with Big Data frameworks: Spark, Databricks, Flink, Beam
- Knowledge of orchestration tools: Airflow, NiFi, AWS Glue, Azure Data Factory, Cloud Composer
- Experience with streaming platforms: Kafka, Kinesis, SQS, Event Hub, Pub/Sub, IoT Hub
- Familiarity with data warehousing and analytics: Snowflake, Redshift, Athena, Synapse
- Experience with containerization & serverless technologies (Docker, Lambda, Fargate, Functions, Cloud Run)
- Understanding of data governance, data quality, and data lifecycle processes
- Strong communication and collaboration skills
- English at Upper-Intermediate level
- Bachelor’s degree in Computer Science or similar (preferred)
- AWS or Snowflake certifications (preferred)
Responsibilities
- Design and optimize distributed computing architectures for scalable Big Data processing
- Build and enhance batch and streaming data pipelines
- Implement advanced data processing: real-time analytics, ML-driven models, predictive insights
- Develop reusable, parameterized ETL/ELT pipelines for DWH and lakehouse solutions
- Conduct end-to-end data flow validation and performance optimization
- Implement best practices for data quality, lineage, and governance
- Configure and maintain CI/CD flows and release pipelines
- Work with containerized and multi-container development environments
- Troubleshoot complex data infrastructure and processing issues
- Contribute to architectural decisions and technology selection
- Ensure secure customer data management throughout the entire project lifecycle
- Follow data retention, deletion, and off-boarding procedures
- Mentor junior engineers and support technical decision-making
Skills
Big Data engineering
SQL
Programming languages (Python, Scala,或
Education
Bachelor’s degree in Computer Science or similar
Tools
AWS
Snowflake
DBT
Hadoop ecosystems (EMR/HDInsight/Dataproc)
Apache Spark/Databricks/Flink/Beam
Airflow/NiFi/AWS Glue/Azure Data Factory/Cloud Composer
Kafka/Kinesis/SQS/Event Hub/Pub/Sub/IoT Hub
Snowflake/Redshift/Athena/Synapse
Docker/Lambda/Fargate/Functions/Cloud Run
Job description
- 5+ years of industrial experience in Big Data Engineering
- Strong proficiency in SQL and at least one programming language: Python, Scala, or Java
- Hands-on experience with AWS and Snowflake
- Strong Experience in Data Build Tool (DBT)
- Experience with cloud storages: AWS S3, Azure Storage, Google Cloud Storage
- Practical skills with Hadoop-like environments (EMR, HDInsight, Dataproc)
- Experience with Big Data frameworks: Apache Spark, Databricks, Apache Flink, Apache Beam
- Knowledge of orchestration tools: Airflow, NiFi, AWS Glue, Azure Data Factory, Cloud Composer
- Experience with streaming platforms: Kafka, Kinesis, SQS, Event Hub, Pub/Sub, IoT Hub
- Familiarity with data warehousing and analytics: Snowflake, Redshift, Athena, Synapse
- Experience working with containerization & serverless technologies (Docker, Lambda, Fargate, Functions, Cloud Run)
- Understanding of data governance, data quality, and data lifecycle processes
- Strong communication and collaboration skills
- English at Upper?Intermediate level
- Bachelor’s degree in Computer Science or similar (preferred)
- AWS or Snowflake certifications (preferred)
AND YOU WANT TO:
- Design and optimize distributed computing architectures for scalable Big Data processing
- Build and enhance batch and streaming data pipelines
- Implement advanced data processing: real?time analytics, ML-driven models, predictive insights
- Develop reusable, parameterized ETL/ELT pipelines for DWH and lakehouse solutions
- Conduct end?to?end data flow validation and performance optimization
- Implement best practices for data quality, lineage, and governance
- Configure and maintain CI/CD flows and release pipelines
- Work with containerized and multi-container development environments
- Troubleshoot complex data infrastructure and processing issues
- Contribute to architectural decisions and technology selection
- Ensure secure customer data management throughout the entire project lifecycle
- Follow data retention, deletion, and off-boarding procedures
- Mentor junior engineers and support technical decision-making
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
