下面我们创建一个真正的爬虫例子
爬取我的博客园个人主页首页的推荐文章列表和地址
scrape_home_articles.py
from urllib.request importurlopenfrom bs4 importBeautifulSoupimportre
html= urlopen("/davidgu")
bsObj= BeautifulSoup(html, "html.parser")for link in bsObj.find("div", {"id":"main_container"}).findAll("a", href=pile("^/davidgu/p")):if 'href' in link.attrs and not('class' inlink.attrs):print(link.string)print(link.attrs['href'])print("--------------------------------------------------------------")
运行结果：
[置顶]解决adb server端口被占用的问题
/davidgu/p/4515236.html
--------------------------------------------------------------
[置顶]解决Eclipse下不自动拷贝apk到模拟器问题( The connection to adb is down, and a sever
/davidgu/p/4390661.html
--------------------------------------------------------------
常用的正则表达式一览
/davidgu/p/4831357.html
--------------------------------------------------------------
C++ 11 - STL - 函数对象(Function Object) (上)
/davidgu/p/4829097.html
--------------------------------------------------------------
...
