Python Developer (Web Crawling Exp)

Python Developer (Web Crawling Exp) - 5+years - Kochi

Responsibilities:

• Develop and maintain efficient web crawlers with a focus on speed, performance, and scalability .
• Write clean, reusable, and optimized Python code for large-scale data extraction.
• Analyze and understand website structures, implement logic to handle dynamic and JS-rendered pages .
• Troubleshoot and debug crawl issues, optimize request handling, and improve crawler reliability.
• Collaborate with backend and data teams to ensure smooth data flow and integration.
• Monitor crawler performance, logs, and resolve runtime errors.
• Participate in code reviews, documentation, and continuous process improvements.

Skills Required:

• Strong coding skills in Python .
• Good understanding of how web pages, HTTP, and crawlers work.
• Hands-on experience with Docker , Postman , and PowerShell .
• Basic understanding of HTML , AngularJS , RabbitMQ , and Celery .
• Experience with Git and Bitbucket for version control and team collaboration.
• Familiarity with XPath , CSS selectors , and regular expressions for content extraction.

Preferred:

• Exposure to Scrapy , Puppeteer/Playwright , or other crawling frameworks.
• Understanding of Elasticsearch , HAProxy , or Curator for large-scale data handling.