mirror of
https://github.com/ivuorinen/cheatsheet-tldr.git
synced 2026-02-04 22:44:00 +00:00
38 lines
891 B
Plaintext
38 lines
891 B
Plaintext
---
|
|
syntax: markdown
|
|
tags: [tldr, common]
|
|
source: https://github.com/tldr-pages/tldr.git
|
|
---
|
|
# scrapy
|
|
|
|
> Web-crawling framework.
|
|
> More information: <https://docs.scrapy.org/en/latest/topics/commands.html#using-the-scrapy-tool>.
|
|
|
|
- Create a project:
|
|
|
|
`scrapy startproject {{project_name}}`
|
|
|
|
- Create a spider (in project directory):
|
|
|
|
`scrapy genspider {{spider_name}} {{website_domain}}`
|
|
|
|
- Edit spider (in project directory):
|
|
|
|
`scrapy edit {{spider_name}}`
|
|
|
|
- Run spider (in project directory):
|
|
|
|
`scrapy crawl {{spider_name}}`
|
|
|
|
- Fetch a webpage as Scrapy sees it and print the source to `stdout`:
|
|
|
|
`scrapy fetch {{url}}`
|
|
|
|
- Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity):
|
|
|
|
`scrapy view {{url}}`
|
|
|
|
- Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available):
|
|
|
|
`scrapy shell {{url}}`
|