GitHub - 451hackathon/legal_crawler

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 28 Commits
censorship_crawler		censorship_crawler
.gitignore		.gitignore
LICENSE		LICENSE
README		README
first.py		first.py
requirements.txt		requirements.txt
second.py		second.py
third.py		third.py

Repository files navigation

Legal Crawler
=============

Installation requirements:

pip -r requirements.txt

Running the crawler

  $ cd censorship_crawler
  $ scrapy crawl 451

Running the crawler against a specific URL:

  $ cd censorship_crawler
  $ scrapy crawl -a url=https://www.example.com 451