Recently I’ve become aware that I don’t usually document my problems or solutions, which usually means I end up searching Google time after time in a desperate attempt to find the same blog or forum post which helped me the last time round. So from now on I’ll start posting the problems and solutions to my own blog, which I imagine will be substantially easier to find!
In my last article I discussed the different techniques that could be used in order to scrape Google’s search engine without being detected. This week we’ll be looking at creating a script to scrape another popular search engine – Bing, using the API which Microsoft provide. It should be noted that Google also provide an API; which allows you to make 100 queries a day (approx 3000 / month); compared to 5000 queries a month with Microsoft Bing.
Using the API is certainly the recommended method for scraping results from either search engine, and the previous article should only be used for education purposes. Wink Wink.