1.問(wèn)題
對(duì)展會(huì)數(shù)據(jù)分類后，我的新任務(wù)是如何通過(guò)公司名肯适、公司地址缔恳、國(guó)家等海關(guān)數(shù)據(jù)推斷出該公司的官網(wǎng)網(wǎng)站（若官網(wǎng)不存在則不考慮）
以下數(shù)據(jù)僅供參考：
| 公司名 | 國(guó)家 | 地址 |
|---|---|---|
| JPW INDUSTRIES INC | 427 NEW SANFORD RD LAVERGNE TN 37086 US | |
| Fujian Xishi Co., Ltd | CN, CHINA | |
| BusinessPartner Co.,ltd | ||
| BENKAI Co.,Ltd | ||
| GOLD INC | 18245 E 40TH AVE AURORA CO 80011 US |
需要得到結(jié)果：
| 公司名 | 官方網(wǎng)站 |
|---|---|
| JPW INDUSTRIES INC | http://http://www.jpwindustries.com/ |
| Fujian Xishi Co., Ltd | http://www.xishigroup.com/ |
| BusinessPartner Co.,ltd | http://www.traderthailand.com/ |
| BENKAI Co.,Ltd | http://www.benkaico.com |
| GOLD INC | https://goldbuginc.com/ |
2.解決
由數(shù)據(jù)可看出，公司名是絕對(duì)存在的娄周，故解決思路是從公司名出發(fā)没卸，而不怎么全面的國(guó)家以及地址信息則用來(lái)提高準(zhǔn)確度。
大體思路是這樣的秒旋，若公司官網(wǎng)存在约计，那么通過(guò)搜索引擎定會(huì)被檢索到，搜索引擎自然首選google迁筛，所以可以先通過(guò)獲取谷歌搜索的結(jié)果煤蚌，然后分析獲取的結(jié)果，從而得出最可能是該公司網(wǎng)站的url细卧。
初步搜索一下尉桩，看看各種情況：
第一種情況，檢索即可很直觀地得出結(jié)果
JPW INDUSTRIES INC
第二種情況贪庙，檢索不能直觀地得出結(jié)果蜘犁，但官網(wǎng)確實(shí)存在（第二檢索個(gè)結(jié)果）
Fujian Xishi Co., Ltd
第三種情況，輸入公司名+公司地址和只輸入公司名得出的結(jié)果不一樣
GOLD INC
對(duì)于第三種情況止邮，可以看出輸入公司名+公司地址得出的結(jié)果是絕對(duì)正確的这橙。
觀察第三種情況，當(dāng)輸入公司名+公司地址時(shí)导披，返回結(jié)果的右側(cè)會(huì)出現(xiàn)公司的詳細(xì)信息屈扎，經(jīng)過(guò)驗(yàn)證，若出現(xiàn)這種情況撩匕，則其website對(duì)應(yīng)的url絕對(duì)正確鹰晨。
故代碼的第一步驟可以首先檢索公司名+公司地址，觀察website元素是否存在，若存在模蜡，返回公司官網(wǎng)漠趁，否則，對(duì)公司名進(jìn)行檢索哩牍。
代碼：
def searchWeb(query, tld='com', lang='en', tbs='0', num=10, safe='off', tpe='', user_agent=None):
query = quote_plus(query)
get_page(url_home % vars())
url = url_search_num % vars()
# Request the Google Search results page.
html = get_page(url)
try:
href = re.findall(r'Directions</a><a\s*class="fl"\s*href="(.*?)".*?>Website', str(html))[0]
link = filter_result(href)
return link
except:
pass
return None
直接返回公司官網(wǎng)
能直接獲取公司官網(wǎng)畢竟是少數(shù)棚潦，大多數(shù)據(jù)還是要通過(guò)一步步計(jì)算得出，主要經(jīng)過(guò)以下步驟：
獲取搜索引擎檢索結(jié)果提取url->初步排除某些url->余弦相似度計(jì)算最可能的結(jié)果
2.1.獲取搜索引擎檢索結(jié)果提取url
對(duì)于谷歌搜索膝昆，我使用了MarioVilas的項(xiàng)目google丸边，畢竟在國(guó)內(nèi)，為了以防萬(wàn)一我也寫(xiě)了yahoo搜索荚孵，代碼如下：
#!/usr/bin/env
# -*-coding:utf-8-*-
# script: yahooSearch.py
__author__ = 'howie'
import sys
import time
if sys.version_info[0] > 2:
from urllib.request import Request, urlopen
from urllib.parse import quote_plus, urlparse
else:
from urllib import quote_plus
from urllib2 import Request, urlopen
from urlparse import urlparse, parse_qs
try:
from bs4 import BeautifulSoup
is_bs4 = True
except ImportError:
from BeautifulSoup import BeautifulSoup
is_bs4 = False
url_search = "https://search.yahoo.com/search?p=%(query)s&b=%(start)s&pz=%(num)s"
headers = {
'accept-encoding': 'gzip, deflate, sdch, br',
'accept-language': 'zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4',
'upgrade-insecure-requests': '1',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'cache-control': 'max-age=0',
'authority': ' search.yahoo.com'
}
# 默認(rèn)user_agent
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
def filter_link(link):
try:
linkData = urlparse(link)
if linkData.netloc and "yahoo.com" not in linkData.netloc and "/search/srpcache" not in linkData.path:
return link
except Exception:
pass
return None
def yahooSearch(query, start=0, num=10, page=1, pause=1.0):
"""
獲取雅虎搜索url
:param query: 搜索關(guān)鍵詞
:param start: 開(kāi)始條目,最好為0
:param num: 搜索條目 建議10的倍數(shù)
:param page: 頁(yè)數(shù)
:param pause: 停頓時(shí)間
:return: 返回url
"""
query = quote_plus(query)
while page > 0:
url = url_search % vars()
time.sleep(pause)
request = Request(url)
request.add_header('User-Agent', user_agent)
response = urlopen(request)
html = response.read()
if is_bs4:
soup = BeautifulSoup(html, 'html.parser')
else:
soup = BeautifulSoup(html)
anchors = soup.find(id="web").find_all('a')
for a in anchors:
try:
link = filter_link(a["href"])
if link:
yield link
except KeyError:
continue
start += num + 1
page -= 1
if __name__ == '__main__':
# GOLD INC 18245 E 40TH AVE AURORA CO 80011 US
for url in yahooSearch("GOLD INC 18245 E 40TH AVE AURORA CO 80011 US"):
print(url)
2.2.初步排除某些url
這個(gè)可根據(jù)個(gè)人需求來(lái)配置妹窖，可添加webConfig.py腳本，排除某些url:
# -*-coding:utf-8-*-
__author__ = 'howie'
config = dict(
# www開(kāi)頭或分割后數(shù)組大于二的網(wǎng)站
forbid_www=["www.linkedin.com", "www.alibaba.com"],
# 非www開(kāi)頭的網(wǎng)站
forbid=["imexbb.com", "made-in-china.com"]
)
以公司名BENKAI Co.,Ltd為例收叶，初步獲取url：
['http://www.benkaico.com', 'http://hisupplier.com', 'http://www.hktdc.com/en']
2.3.余弦相似度計(jì)算最可能的結(jié)果
對(duì)于公司BENKAI Co.,Ltd骄呼，我們獲得了三個(gè)結(jié)果，現(xiàn)在又該如何從該列表中取得最可能的結(jié)果呢判没。
這里可以采用余弦相似度蜓萄，具體公式可google，稍稍解釋下：
對(duì)于這三個(gè)網(wǎng)站：
['http://www.benkaico.com', 'http://hisupplier.com', 'http://www.hktdc.com/en']```
可以通過(guò)計(jì)算各個(gè)網(wǎng)站的`title`和`BENKAI Co.,Ltd`的相似程度來(lái)取得最可能的結(jié)果澄峰。
#####2.3.1：對(duì)各網(wǎng)站title進(jìn)行分詞
```python
{'http://www.benkaico.com': ['benkai', 'co.', ',', 'ltd', '.'],
'http://www.hktdc.com/en': ['hktdc.com', 'a\x80\x93', 'page', 'not', 'found'],
'http://hisupplier.com': ['china', 'suppliers', ',', 'suppliers', 'directory', ',', 'china', 'manufacturers', 'directory', '-', 'hisupplier.com']}
2.3.2：構(gòu)建單詞向量
{'http://www.benkaico.com': [[0, 1, 1, 1, 1], [1, 1, 1, 1, 1]],
'http://www.hktdc.com/en': [[1, 1, 0, 0, 1, 0, 0, 0, 1], [0, 0, 1, 1, 0, 1, 1, 1, 0]],
'http://hisupplier.com': [[0, 0, 0, 1, 0, 0, 1, 1, 0, 1], [2, 1, 1, 0, 2, 1, 2, 0, 2, 0]]}
2.3.3:計(jì)算余弦相似度
{'http://www.benkaico.com': 0.94427190999915878,
'http://www.hktdc.com/en': 0.0,
'http://hisupplier.com': 0.31451985913875646}
通過(guò)比較嫉沽，可以看到http://www.benkaico.com相似度最高的結(jié)果，跟真實(shí)結(jié)果一樣俏竞。
http://www.benkaico.com
全部步驟代碼如下：
# -*-coding:utf-8-*-
__author__ = 'howie'
from urllib import parse
from bs4 import BeautifulSoup
from collections import defaultdict
import requests
import re
import nltk
import time
from config.webConfig import config
from CosineSimilarity import CosineSimilarity
from search.yahooSearch import yahooSearch
from search.gooSearch import search, searchWeb
class Website(object):
"""
通過(guò)公司名等信息獲取網(wǎng)站官網(wǎng)
"""
def __init__(self, engine='google'):
self.forbid_www = config["forbid_www"]
self.forbid = config["forbid"]
self.engine = engine
def get_web(self, query, address=""):
"""
獲取域名
:param query: 搜索詞
:param address: 在此加上地址時(shí)绸硕，query最好是公司名
:return: 返回最可能是官網(wǎng)的網(wǎng)站
"""
if self.engine == 'google' and address:
allQuery = query + " " + address
result = searchWeb(query=allQuery, num=5)
if result:
return result
allDomain = self.get_domain(query)
if len(allDomain) == "1":
website = allDomain[0]
else:
# 初步判斷網(wǎng)站域名
counts = self.get_counts(allDomain)
largest = max(zip(counts.values(), counts.keys()))
if largest[0] > len(allDomain) / 2:
website = largest[1]
else:
# 獲取對(duì)應(yīng)域名標(biāo)題
domainData = self.get_title(set(allDomain))
# 計(jì)算相似度
initQuery = nltk.word_tokenize(query.lower(), language='english')
# 余弦相似性計(jì)算相似度
cos = CosineSimilarity(initQuery, domainData)
wordVector = cos.create_vector()
resultDic = cos.calculate(wordVector)
website = cos.get_website(resultDic)
return website
def get_domain(self, query):
"""
獲取谷歌搜索后的域名
:param query:搜索條件
:return:域名列表
"""
allDomain = []
if self.engine == "google":
for url in search(query, num=5, stop=1):
allDomain += self.parse_url(url)
elif self.engine == "yahoo":
for url in yahooSearch(query):
allDomain += self.parse_url(url)
if not allDomain:
allDomain.append('')
return allDomain
def parse_url(self, url):
allDomain = []
domainParse = parse.urlparse(url)
# 英文網(wǎng)站獲取
if "en" in domainParse[2].lower().split('/'):
domain = domainParse[1] + "/en"
else:
domain = domainParse[1]
domainList = domain.split('.')
# 排除干擾網(wǎng)站
if len(domainList) >= 3 and domainList[0] != "www":
isUrl = ".".join(domain.split('.')[-2:])
if isUrl not in self.forbid:
allDomain.append(domainParse[0] + "://" + isUrl)
elif domain not in self.forbid_www:
allDomain.append(domainParse[0] + "://" + domain)
return allDomain
def get_title(self, setDomain):
"""
獲取對(duì)應(yīng)網(wǎng)站title，并進(jìn)行分詞
:param allDomain: 網(wǎng)站集合
:return: 網(wǎng)站:title分詞結(jié)果
"""
domainData = {}
for domain in setDomain:
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, sdch",
"Accept-Language": "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4",
"Cache-Control": "max-age=0",
"Proxy-Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36",
}
try:
data = requests.get(domain, headers=headers).text
soup = BeautifulSoup(data, 'html.parser')
title = soup.title.get_text()
title = re.sub(r'\r\n', '', title.strip())
titleToken = nltk.word_tokenize(title.lower(), language='english')
domainData[domain] = titleToken
except:
pass
return domainData
def get_counts(self, allDomain):
"""
返回網(wǎng)站列表各個(gè)域名數(shù)量
:param allDomain: 網(wǎng)站列表
:return: 網(wǎng)站:數(shù)量
"""
counts = defaultdict(int)
for eachDomain in allDomain:
counts[eachDomain] += 1
return counts
if __name__ == '__main__':
# allQuery = ["National Sales Company, Inc.", "Decor Music Inc.","Fujian Xishi Co., Ltd","Kiho USA Inc.","BusinessPartner Co.,ltd","BENKAI Co.,Ltd"]
# GOLD INC 18245 E 40TH AVE AURORA CO 80011 US
allQuery = ["ALZARKI INTERNATIONAL"]
website = Website(engine='google')
for query in allQuery:
time.sleep(2)
website = website.get_web(query=query)
print(website)
計(jì)算余弦相似度代碼
# -*-coding:utf-8-*-
# script: CosineSimilarity.py
__author__ = 'howie'
import numpy as np
from functools import reduce
from math import sqrt
class CosineSimilarity(object):
"""
余弦相似性計(jì)算相似度
"""
def __init__(self, initQuery, domainData):
self.title = initQuery
self.data = domainData
def create_vector(self):
"""
創(chuàng)建單詞向量
:return: wordVector = {} 目標(biāo)標(biāo)題以及各個(gè)網(wǎng)站標(biāo)題對(duì)應(yīng)的單詞向量
"""
wordVector = {}
for web, value in self.data.items():
wordVector[web] = []
titleVector, valueVector = [], []
allWord = set(self.title + value)
for eachWord in allWord:
titleNum = self.title.count(eachWord)
valueNum = value.count(eachWord)
titleVector.append(titleNum)
valueVector.append(valueNum)
wordVector[web].append(titleVector)
wordVector[web].append(valueVector)
return wordVector
def calculate(self, wordVector):
"""
計(jì)算余弦相似度
:param wordVector: wordVector = {} 目標(biāo)標(biāo)題以及各個(gè)網(wǎng)站標(biāo)題對(duì)應(yīng)的單詞向量
:return: 返回各個(gè)網(wǎng)站相似度值
"""
resultDic = {}
for web, value in wordVector.items():
valueArr = np.array(value)
# 余弦相似性
squares = []
numerator = reduce(lambda x, y: x + y, valueArr[0] * valueArr[1])
square_title, square_data = 0.0, 0.0
for num in range(len(valueArr[0])):
square_title += pow(valueArr[0][num], 2)
square_data += pow(valueArr[1][num], 2)
squares.append(sqrt(square_title))
squares.append(sqrt(square_data))
sum_of_squares = reduce(lambda x, y: x * y, squares)
resultDic[web] = numerator / sum_of_squares
return resultDic
def get_website(self, resultDic):
"""
獲取最可能是官網(wǎng)的網(wǎng)站
:param resultDic: 各個(gè)網(wǎng)站相似度值
:return: 最可能的網(wǎng)站 也可能為空
"""
website = ''
largest = max(zip(resultDic.values(), resultDic.keys()))
if largest[0]:
website = largest[1]
# 當(dāng)相似度為0
else:
websites = [key for key, values in resultDic.items() if values == 0.0]
for eachWebsite in websites:
keyword = ','.join(self.data[eachWebsite]).lower()
if 'home' in keyword or "welcome" in keyword:
website = eachWebsite
return website
3.總結(jié)
至此魂毁，若有更好的解決方案玻佩，歡迎賜教，謝謝席楚。
