fetching only website details as search engine does

I have to fetch website details as search engine does. I need the description of the site,link 开发者_StackOverflow中文版and some info about them and will store it in my DB. Is there any libraries available for doing this? Please remember I can crawl a whole webpage but I need only the information in the format crawled by search engines.

Thanks,

Karthik

---

Which language? APIs and bindings exist for reading webpage content. Do you realize the scale of the task if you wish to create a new 'search engine'? Your question is so generic, there's not a lot of advice that can be given, other than:

Respect robots.txt

Don't hammer the server with requests, you'll soon get your IP blocked by sensible sysadmins.