Importing and Exporting additional start URLs

Top  Previous  Next

In spider mode, you can import additional spider URLs from a text file by clicking on the Import button and specifying its filename. The following is an example of a text file that can be imported by this feature.

http://news.mysite.com/

http://www.myalternativesite.com/

http://www.mysite.com/spider_sitemap.html

http://www.friends_site.com/

 

These URLs will be imported into the list and given the default option of “index and follow”.

It can also import from a list of URLs with comma-separated values to specify the indexing option of that URL. The format for this advanced import list is as follows:

<URL>, <URL option>, [<base URL> [, LIMIT=<limit>] [, WEIGHT=<weight>] ]

 

The following is an example list in the above specified format:

http://news.mysite.com/, INDEX_AND_FOLLOW

http://www.myalternativesite.com/, INDEX_AND_FOLLOW

http://www.othersite.com/, INDEX_AND_FOLLOW, http://pages.othersite.com/

http://www.mysite.com/spider_sitemap.html, FOLLOW_ONLY

http://www.myschool.edu/bob/index.html, INDEX_AND_FOLLOW, http://www.myschool.edu/, LIMIT=50, WEIGHT=-2

http://www.friends_site.com/, INDEX_ONLY

 

The URL options available correspond to the “Advanced Spider URL options” described in section 2.1.4 as follows:

INDEX_AND_FOLLOW: Index page and follow links (default)
INDEX_AND_FOLLOW_ALL: Index page and follow all links
INDEX_ONLY: Index single page only
FOLLOW_ONLY: Follow links only
FOLLOW_ALL: Follow all links on this page only

See "Start spider URL" for more details on each of these options.

The LIMIT= flag must be followed by a value specifying a restriction on the maximum number of files to index for a particular start point. This is the same setting as the one which you can specify on the "Advanced Spider URL options" window.

The WEIGHT= flag must be followed by a value which specifies the boosting or indexing weighting for the pages under this start point. This should be a value between +5 and -5 similar to the PAGEBOOST meta value. It is the equivalent of specifying PAGEBOOST meta tags for each of the individual pages that are indexed  under the start point in question. See "Weightings" for more information.

You can also export your current list of URLs to a CSV text file by way of the Export button, which will export in the latter layout as given in the above example.