Crawler
Crawler walks the pages of a site and takes the files it finds on them. This page covers the sites it knows by name, the kinds of page it reads anywhere else, where it may go and what it keeps, when it uses a browser, how it treats a site, and pages you look at before a run.
Sites it knows by name
For a few sites Crawler has a recipe: a description of where the site keeps its files and how to page through them. On these sites the wizard selects Crawler, and nothing is asked about the kind of page.
| Site | What Crawler takes there |
|---|---|
| 500px | a photographer, a feed such as the popular photos, a gallery, a photo story or one photo; the long side up to 2048 pixels |
| Bing Images | image search results |
| Flickr | a photostream, an album, favourites, a group pool, a tag, Explore, one photo or a search; the original only where the author allows downloads, otherwise the largest size there is |
| Google Images | image search results |
| IMDb | a film, a person or a gallery |
| a subreddit or a post |
Bing Images and Google Images are read through a browser. Before a run on Google Images, the search opens for you to look at - see Pages you look at before a run.
For IMDb the wizard asks Download images and Download videos; IMDb hands over at most 100 videos. It also shows IMDb states conditions for the use of its data., and the project can be created only once you tick I have read this.
For Reddit the wizard shows Reddit application id, already filled in. Reddit answers only to a registered application; if you hit its limits, register your own at reddit.com/prefs/apps as an installed app and enter its id in Reddit application id.
Any other page
On a site it does not know by name, Crawler's card says Works on any page, but knows nothing particular about this site. Pick Crawler, and the wizard asks Where do the files come from? Nothing is selected in advance: only you know what the page is.
| Choice | What it means |
|---|---|
| Just this page | Files linked from the address you gave, and nothing further |
| Gallery or album | A page with a set of images on it |
| Forum thread | One topic, with attachments and pictures in the posts |
| The whole site, or a part of it | By default only the address you gave and what sits under it |
Then File types asks what to Download: Pictures, ticked by default, Video, Audio, Documents and Archives. A file whose kind neither its address nor the site's answer gives away is taken anyway.
Apart from Where may this go?, below, the wizard asks nothing else; the rest of each choice's settings is in the project's Site settings.
Where it may go
For The whole site, or a part of it, the wizard also asks Where may this go? It lists areas from the narrowest to the widest, and the job takes in whatever the pattern on the line you pick covers. For https://example.com/gallery/7/ the lines are example.com/gallery/7/*, example.com/gallery/*, example.com/* and *.example.com/* - the whole site with its subdomains.
Addresses in the project's Site settings has the rest:
- Additional addresses - one address pattern per line, such as
site.com/gallery/*. Links matching them belong to the job as well; the line picked in the wizard is kept here as the first. - Exceptions - patterns of addresses the walk does not go to, however it got pointed at them. They win over the addresses above.
- Steps from the start - how far the walk may go from the address: one step for every link it follows, 3 by default and 20 at most. Another page of the same listing costs nothing, and neither does a file.
- External links - links that lead out of the job: Ignore them, Only take files from them, which is the default, or Follow the pages and take the files.
- Allowed external addresses - where the walk may step out to;
*, the default, is anywhere. - Which addresses are files - one rule per line, such as
site.com/get-image/* = pictures, for a script that serves a picture, or for a download link with no extension. - Rules for following - your own conditions for a link, as Do not go there when or Go there only when.
Keep the site's own folders, under Folders, files everything the way the site does - the host first, then the folders of the address. It is off by default, and everything lands together.
What it keeps
Filters in the project's Site settings:
- Minimum size and Maximum size, in bytes; 0 means no limit. A file whose size the site did not state is taken. For Forum thread the minimum size starts at 20480 bytes: in a post, avatars, smilies and rank badges are ordinary pictures too.
- Minimum width and Minimum height, in pixels. A picture's size is inside the file, so this is decided after it has been fetched: a picture that does not fit is deleted, and the journal says so.
- Rules for keeping - your own conditions for a file, as Do not take it when.
For Just this page, Filters also has Which addresses are files: nothing is followed from that page, so an address that gives away its kind only in the site's answer is never asked for.
For Gallery or album, Paging has Pages of one listing: how many pages of the same listing to walk. 0, the default, is all of them - right for an album and wrong for an endless feed.
When it uses a browser
Some sites draw their pages with script, and without a browser there is nothing on them to find. Use a browser, under Browser in the project's Site settings, is Decide automatically, Yes or No. Left to decide, Crawler reads the first page of each site both ways and keeps whichever found more; the journal says which, with both numbers. A browser is tens of times slower.
When a page gives nothing, the journal says Nothing found on … - the page may need a browser.
Manners
Manners, in the project's Plugin settings, is how Crawler treats a site:
- Seconds between requests - the pause between any two requests of a run, whichever sites they go to, 1 by default and 60 at most. The shorter it is, the likelier the site is to start refusing or to close its doors to you; 0 is the fastest way to have your IP address banned. See The pace a run keeps.
- Keep the pages - stores each page that was read, so that the recipe editor can repeat a run over them without touching the site again. On by default.
- Use recipes - on a site Crawler knows by name, it follows the site's recipe. Turn it off to walk the site the ordinary way. On by default.
- Recognise image hosts - reads a link to an image host as a picture with one hop in front of it. Worth turning off if a host has changed its pages and an advertisement arrives instead of a picture. On by default.
Crawler fetches two files at a time, counting all sites together. When a site asks it to slow down, it does, and the journal says … asked us to slow down, easing off. Seconds: ….
Pages you look at before a run
Some sites need you to do something by hand first: sign in, pass a check or narrow a search. Show the page before a run, also under Manners, decides when the starting page opens:
- Never - the run starts at once.
- When the site needs it - the default. The page opens on a site whose recipe says it needs you, such as Google Images.
- Always - the page opens for every address of the project, and once for a pattern, at its first address.
The page opens in a window called Check this before the download starts and waits. Do what the site needs, then click Continue: the run starts from wherever you ended up, so on a search it keeps both the query and the filters you set. Cancel stops the run before it starts.
Ticking This site needs me to sign in first on the wizard's last step sets Always.
Fix the parser
On the Properties tab of a project, in the NeoDownloader window, a card offers Fix the parser: it opens the recipe editor on the project's last run, at the step that stopped finding anything. The editor is experimental. Before it opens, NeoDownloader warns The recipe editor is not finished and says that the editor may not manage to repair the run.