74 Star 116 Fork 42

calvinwilliams / simspider

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
ChangeLog-EN 4.08 KB
一键复制 编辑 原始数据 按行查看 历史
calvinwilliams 提交于 2016-04-02 14:19 . UPDATE TO V2.6.7
ver 2.6.7 2016-04-02 calvin
* Fixed something warn about libcurl call interface
ver 2.6.6 2015-09-23 calvin
* Fixed some bugs
ver 2.6.5 2015-04-09 calvin
* Optimize memory used
* Adjust makefile*
ver 2.6.4 2015-03-30 calvin
* Write doc/simspider.vsd
ver 2.6.3 2015-03-29 calvin
* Add callback function funcBeginTaskProc and SetBeginTaskProc for setting , add return MACRO SIMSPIDER_INFO_IGNORE_THIS_TASK for ignore the URL
* Modify some param type 'long' to 'int'
ver 2.6.2 2015-03-28 calvin
* Fixed some bugs
* Add function GetCurlStillRunning for getting still running request count in callback function
* Add function GetCurlFinishedCount for getting finished count after call SimSpiderGo
* Add return MACRO to callback function
ver 2.6.1 2015-03-27 calvin
* Fixed some bugs
ver 2.6.0 2015-03-26 calvin
* depart request queue and done queue implemention for default
* Remove callback function funcTravelDoneQueueProc
ver 2.5.3 2015-03-24 calvin
* Fixed a problem about not support CURLOPT_ACCEPT_ENCODING and CURLOPT_TRANSFER_ENCODING on low-version libcurl by use condition macro
* Fixed a bug in function ResetSimSpiderEnv with multiplexing concurrency
ver 2.5.2 2015-03-23 calvin
* Add function SetMaxRetryCount for setting max retry count
* Add function GetDoneQueueUnitRetryCount for getting retry count at last
* Add function SetAcceptEncoding for setting http accept-encoding
* Add function SetTransferEncoding for setting http transfer-encoding
* Add libcurl's DNS Cache
* Swap HtmlLinkParser and pfuncResponseBodyProc
* Improve the main process framework of events for performance
ver 2.5.1 2015-03-16 calvin
* Solve a problem of compilation
* Add function 'MoveUnsuccessfulDoneQueueUnitsToRequestQueue','ResetDoneQueue' used to move back to request queue from done queue
ver 2.5.0 2015-03-14 calvin
* Adjusted function FuncTravelDoneQueueProc prototype
* Add request queue and queue callback mechanism, the user can alternative the two queue using database or Redis and so on ( experimental )
* Fixed a problem with url contains multibyte-characters
ver 2.4.0 2015-03-13 calvin
* Add multiplexing concurrency automatic adjustment mechanism
* Add HTTP request header custom special list
* Append request header 'Referer' automatically
* Improve the main process framework of events for performance
* The curl object reuse mechanism for performance
* Adjust the sample crawler 'simspider' code
ver 2.3.0 2015-03-08 calvin
* Add the error code SIMSPIDER_ERROR_FUNCPROC_INTERRUPT, if callback function returns the code, crawling process interrupt immediately
* Modify the function AppendRequestQueue prototypes
* Modify the function SimSpiderGo prototypes
* Add function SetSimSpiderPublicDataPtrGetSimSpiderPublicDataPtr , Used to pass the crawler engine public data
* Add function etDoneQueueUnitPrivateDataPtrGetDoneQueueUnitPrivateDataPtr , Used to pass private data when crawl each url
* Add function EnableHtmlLinkerParser , used to set the crawler engine internal quick HTML links parser enabled, the default is enabled
* Fixed a concurrent problem
* Optimized the crawler example program 'simspider'
ver 2.2.1 2015-02-09 calvin
* Fixed the automatic completion bug from entrance to website
* Fixed a safe pointer bug
* Fixed a bug on WINDOWS platform
ver 2.2.0 2015-02-07 calvin
* Modify the function SimSpiderGo prototypes. Well, I don't change since
* Modify the function name AppendRequestUnit to AppendRequestQueue. Well, I don't change since too
* add function ResizeRequestQueue used to adjust the request queue size, in order to adapt to crawl
* Simplify the util module, remove the third-party libraries iconv dependencies
* Rearrange the error code
ver 2.1.1 2015-02-05 calvin
* Stripping of third-party libraries fasterxml,fasterjson dependencies
ver 2.0.1 2015-02-04 calvin
* Fixed a bug
* WINDOWS Supported
ver 2.0.0 2015-02-01 calvin
* support concurrents in single process
ver 1.0.0 2014-12-12 calvin
* create
C
1
https://gitee.com/calvinwilliams/simspider.git
git@gitee.com:calvinwilliams/simspider.git
calvinwilliams
simspider
simspider
master

搜索帮助