【6】网络与网页-2-爬虫--selenium(模拟网页操作)

一、简介

二、安装

  1. 系统:centos 7.x
  2. python:3.6.3
  3. google-chrome: 7.4
  4. chromedriver: 7.4

下载google-chrome

因geckodriver和firefox因为版本对应关系不清楚,geckodriver没法驱动firefox,所以弃坑

wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

sudo yum localinstall google-chrome-stable_current_x86_64.rpm

yum upgrade google-chrome-stable

google-chrome

pip3 install selenium

下载驱动

cd /data/software
wget -c https://chromedriver.storage.googleapis.com/74.0.3729.6/chromedriver_linux64.zip
unzip  chromedriver_linux64.zip
ln -s /data/software/chromedriver /usr/bin/chromedriver

安装运行依赖

yum install gtk3 gtk3-devel
yum install Xvfb libXfont xorg-x11-fonts*
pip3 install pyvirtualdisplay

三、语法说明

3.1 打开一个页面

driver.get("http://www.google.com")

3.2 获取对象

比如:

<input type="text" name="passwd" id="passwd-id" />

可以这么来获取这个对象

element = driver.find_element_by_id("passwd-id")
element = driver.find_element_by_name("passwd")
element = driver.find_element_by_xpath("//input[@id='passwd-id']")

给对象填充内容

element.send_keys("some text")

# 你还可以通过”Keys”类来模式输入方向键:
element.send_keys(" and some", Keys.ARROW_DOWN)

清除对象的内容

element.clear()

显示网页内容

print(driver.page_source)

四、案例

4.1 序列预测

# coding:utf-8
import time
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import selenium
from selenium import webdriver

result_dir = 'xml_tmp'
if os.path.exists(result_dir):
    os.system('rm -fr %s' % result_dir)
safe_mkdir(result_dir)

#设置流浪器配置,
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--headless')  # 无头模式


driver = webdriver.Chrome(chrome_options=chrome_options)  #打开网页

driver.get("** 登陆网址** ")

#用户登陆
user_name = driver.find_element_by_name("userName")
user_passwd = driver.find_element_by_xpath("//input[@name='password'][@class='login password-field enterFind']")

user_name.clear()
user_name.send_keys("****user")

user_passwd.clear()
user_passwd.send_keys("*** passwd")

#选择预测的界面
login_button = driver.find_element_by_xpath("//button[@onclick='check();']")
login_button.click()

prediction_button = driver.find_element_by_xpath(
    "//a[@class='javaHMl'][@title='**********这里需要替换*****']")

prediction_button.click()

for one_fasta in fastas:
    one_seq = str(one_fasta.seq)
    seq_name = str(one_fasta.id)


    result_dir_2 = os.path.join(result_dir, seq_name)

    ##提交序列
    elem = driver.find_element_by_name("queryTextArea")
    elem.clear()
    elem.send_keys(one_seq)

    run_button = driver.find_element_by_xpath("//button[@class='btn btn-primary'][@onclick='run()']")
    run_button.click()

    time.sleep(2)

    ## 保存结果
    save_button = driver.find_element_by_xpath("//button[@class='btn'][@onclick='save()']")
    save_button.click()

    time.sleep(2)



    ## 去掉另存为的弹框
    driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
    params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': result_dir_2}}
    command_result = driver.execute("send_command", params)

    ### 选择另存为的数据类型
    xml_result = driver.find_element_by_xpath("//a[@class='layui-layer-btn1']")
    xml_result.click()
    time.sleep(5)

driver.close()

五、报错

5.1 报错

selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilitie

报错原因:

selenium 、firefox、geckodriver版本不兼容

版本一直没试对,干脆换了一个浏览器

5.2 报错2 DevToolsActivePort file doesn’t exist

selenium调用无头chrome浏览器进行爬取的Python程序报出如下的错误,解决办法

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')  # 不用打开图形界面
chrome_options.add_argument('--no-sandbox')  # 让chrome在root权限下跑
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-dev-shm-usage')

六、讨论

6.1 chrome_options中涉及到哪些参数以及对应含义

https://cs.chromium.org/chromium/src/chrome/common/pref_names.cc

6.2 去掉另存为的弹框

driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': result_dir_2}}
command_result = driver.execute("send_command", params)

### 选择另存为的数据类型
xml_result = driver.find_element_by_xpath("//a[@class='layui-layer-btn1']")
xml_result.click()

6.3 上传文件(input类型)

fileupload.html中用于上传file的控件

 <input type="file" name="file" id="file" />

众所周知,input标签是可以直接send_keys的,这种情况下就简单粗暴(专一 一点,这种已经就很好了),来看代码示例:

from selenium import webdriver
import time

driver = webdriver.Firefox()
driver.get('http://localhost/test/fileupload.html')
upload = driver.find_element_by_id('file')
time.sleep(12)
upload.send_keys('d:\\all_money.wmv')  # send_keys
print (upload.get_attribute('value'))  # check value

driver.quit()

控制台结果输出:
"C:\Program Files\Python35-32\python.exe" D:/Python/Demo/fileupload.py
all_money.wmv

参考资料

个人公众号,比较懒,很少更新,可以在上面提问题,如果回复不及时,可发邮件给我: tiehan@sina.cn

Sam avatar
About Sam
专注生物信息 专注转化医学