Start spider URL (spider mode only) |
![]() ![]() ![]() |
In spider mode, you are required to specify the URL from which the indexer will start the spider scanning from. Typically, you would point this to the entrance page of your website, (such as index.html) so that it will be able to find links to other pages on your website by following the links it finds on each page (as a visitor would). Also note that the spider indexing mode automatically skip links to external web sites, i.e. those that are outside of the base URL defined (see below). This is to prevent indexing pages outside of the specified website. Advanced spider URL options: Clicking on the Spidering options With each spider URL in this list you can specify the following options:
You can also override the automatic base URL determined from this window, if necessary.
Limits files for this start point You can limit the number of files to index from this particular start point by checking this option. You can specify a global limit for all start points on the "Limits" tab of the Configuration window. Note that when both the global and individual limit is set, both settings will apply, so which ever limit is first reached (ie: the lower limit of the two), will cause the indexer to stop indexing the current start point. Weighting for this start point This adjusts the score weighting for the pages indexed under this start point. This can be used to make pages found from a particular start point or domain to be ranked higher or considered more important than pages from other start points. See "Weightings" for more information. Import and export start points You can also Import and Export additional URLs from a text file using the Import and Export button. See "Importing and Exporting additional start URLs" for more information. The number of start points you can have in this list are only limited by the system resources available. However, the total number of pages indexed would still be limited by the indexing limits (max. pages, max. unique words, etc.) specified on the "Limits" tab. |