1 Star 0 Fork 3.7K

zhao_god / spider-flow

forked from ssssssss-team / spider-flow
Closed
 
Create your Gitee Account
Explore and code with more than 12 million developers,Free private repositories !:)
Sign up
Clone or Download
contribute
Sync branch
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README
MIT

介绍 | 特性 | 插件 | DEMO站点 | 文档 | 更新日志 | 截图 | 其它开源 | 免责声明

介绍

平台以流程图的方式定义爬虫,是一个高度灵活可配置的爬虫平台

特性

  • 支持Xpath/JsonPath/css选择器/正则提取/混搭提取
  • 支持JSON/XML/二进制格式
  • 支持多数据源、SQL select/selectInt/selectOne/insert/update/delete
  • 支持爬取JS动态渲染(或ajax)的页面
  • 支持代理
  • 支持自动保存至数据库/文件
  • 常用字符串、日期、文件、加解密等函数
  • 支持插件扩展(自定义执行器,自定义方法)
  • 任务监控,任务日志
  • 支持HTTP接口
  • 支持Cookie自动管理
  • 支持自定义函数

插件

项目部分截图

爬虫列表

爬虫列表

爬虫测试

爬虫测试

Debug

Debug

日志

日志

其它开源项目

免责声明

请勿将spider-flow应用到任何可能会违反法律规定和道德约束的工作中,请友善使用spider-flow,遵守蜘蛛协议,不要将spider-flow用于任何非法用途。如您选择使用spider-flow即代表您遵守此协议,作者不承担任何由于您违反此协议带来任何的法律风险和损失,一切后果由您承担。

MIT License Copyright (c) 2019 小东 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

About

新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。 expand collapse
Java
MIT
Cancel

Releases

No release

Contributors

All

Activities

Load More
can not load any more
Java
1
https://gitee.com/zhao_god/spider-flow.git
git@gitee.com:zhao_god/spider-flow.git
zhao_god
spider-flow
spider-flow
master

Search

14c37bed 8189591 565d56ea 8189591