nutch 0.9二次开发--抓不到包含?等符号的URL
问题：使用nutch抓取不到http://www.tianya.cn/new/TianyaCity/content.asp?idItem=296&idArticle=53561&idWriter=0&key=0等URL
分析：使用nutch默认的配置过滤文件的话，是不抓取到包含?*!@=等字符的URL
解决办法：修改crawl-urlfilter的过滤规则，
# The url filter file used by the crawl command.
# Better for intranet crawling.# Be sure to change MY.DOMAIN.NAME to your domain name.
# Each non-comment, non-blank line contains a regular expression# prefixed by '+' or '-'. The first matching pattern in the file# determines whether a URL is included or ignored. If no pattern# matches, the URL is ignored.
# skip file:, ftp:, & mailto: urls-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.-[?*!@=] //表示过滤包含指定字符的URL，改为： -[~]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops-.*(/.+?)/.*?\1/.*?\1/
# accept hosts in MY.DOMAIN.NAME +^http://([a-z0-9]*\.)*tianya.cn/[\s\S]* // 过滤正则表达式，([a-z0-9]*\.)*表示任意数字和字母，[\s\S]*表示任意字符
# skip everything else-.
相关资源：敏捷开发V1.0.pptx