In this picture you can see that we have chosen a webpage. Also, how far we want our spider software to look into our website. Maximum depth of 3 directories and the maximum number of files to look for is 10.
We do not want the search engine to go too far and become confused with what you are promoting. Hence, we select a spider depth
and files that relate to mostly what your website contains.