sem爬虫需求

Ben 5e34cceace config.json: add num_threads、task_queue_len		1 년 전
.gitignore	aba9e4e521 first commit	1 년 전
README.md	ea5d771230 use get_recommender_pages instead of get_related_pages	1 년 전
config.json	5e34cceace config.json: add num_threads、task_queue_len	1 년 전
paper2000.json	aba9e4e521 first commit	1 년 전
paper_spider.papers_data.json	9033cbb0f2 updated papers_data.json	1 년 전
requestments.txt	aba9e4e521 first commit	1 년 전
spider.py	5e34cceace config.json: add num_threads、task_queue_len	1 년 전

sem-spider

实现步骤

1、读取配置文件中的数据库参数；

2、连接数据库，并创建 papers 集合；

3、实现 /add_paper 端点，用于添加样本数据；

4、实现 /crawl_data 端点，用于爬取数据；

5、实现 worker 函数，用于处理爬取任务。

APIs

/graph/v1/paper/{paper_id}
/graph/v1/paper/{paper_id}/citations
/graph/v1/paper/{paper_id}/references
# https://api.semanticscholar.org/api-docs/#tag/Paper-Data/operation/post_graph_get_papers

例子：

curl --location 'https://api.semanticscholar.org/graph/v1/paper/61822dc4ea365e1499fbdae7958aa317ad78f39f?fields=title%2CpaperId%2CreferenceCount&limit=100&offset=' \
--header 'x-api-key: B4YUQrO6w07Zyx9LN8V3p5Lg0WrrGDK520fWJfYD'

~~相关论文 related-papers：~~(web接口存在人机验证)

curl --location 'https://www.semanticscholar.org/api/1/paper/61822dc4ea365e1499fbdae7958aa317ad78f39f/related-papers?limit=15&recommenderType=relatedPapers' \
--header 'Cookie: tid=rBIABmR91dK7TwAJJIRdAg=='

推荐论文

https://api.semanticscholar.org/recommendations/v1/papers/forpaper/61822dc4ea365e1499fbdae7958aa317ad78f39f?fields=url,abstract,authors

数据字典

{
    "_id": {
        "$oid": "64796d23c0a763eba149940a"
    },
    "paper_id":3658586,
    "year": "年份",
    "title": "文章标题",
    "slug": "标语",
    "tldr": "概述",
    "numCiting": "被参考次数",
    "numCitedBy": "被引用次数",
    "bages": "",
    "authors": "作者",
    "corpusid": 96048797,
    "url": "...",
	"citations": [],
	"references": [],
	"relaterPapers": [],
}

依赖

requirements.txt

requests
pymongo

运行方式

add_paper

python3 spider.py add_paper --path paper2000.json

crawl_data

python spider.py crawl_data

README.md

sem-spider

实现步骤

APIs

数据字典

依赖

运行方式