Skip to content

glwbr/crawler

Repository files navigation

Basic Usage

  1. Clone the repot to your machine
  2. Make sure that you have nodejs installed
  3. On the project folder run npm install
  4. Call node pointing to main.js and pass a URL to it

Example

node main.js https://www.example.com/

Limitations

  1. It'll take a long time to run on websites like stackoverflow and at somepoint it'll get a 429 - Too many requests exception but it works :).
  2. It doesn't handle links of external domains
  3. Probably many others...

About

One of the crawlers ever created.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published