Donate SIGN UP

adding robots.txt to website?

Avatar Image
bossychloe | 14:45 Thu 05th Mar 2009 | Technology
10 Answers
Hi
Can anyone tell me where and how I add the robots.txt file when working within a 'Web Page Maker' page.
I have produced a file entitled robots.txt and uploaded it (which contains the: User-agent and Disallow script) but when I test it with the Google Analyze robots.txt it still states my web pages are still 'Allowed'.
Please can anyone help me?

Answers

1 to 10 of 10rss feed

Best Answer

No best answer has yet been selected by bossychloe. Once a best answer has been selected, it will be shown here.

For more on marking an answer as the "Best Answer", please visit our FAQ.
it goes in the top level/root of the web site (i.e. where your main index.html is) but will only work if you have your own domain.
oh and what have you put into the txt file?? to exclude your whole site from every search engine it should have in it

User-agent: *
Disallow: /
Question Author
ChuckFickens: When I attempt to change the word 'index.html' to 'robots.txt' a message appears telling me it is not allowed.

Yes I have:
User-agent: *
Disallow: /

So does that mean that I cannot add the robots.txt to my site?

Regards x
I don't understand what you mean by When I attempt to change the word 'index.html' to 'robots.txt' a message appears telling me it is not allowed.

you should just be uploading a new file called robots.txt to your webspace that will sit in the same folder location as your index.html.

Question Author
Sorry I should have explained,.. I am using a 'Web Page Maker' construction software to upload my website.
Question Author
ChuckFickins: I have used the Ipswich WS-FTP to upload my robots.txt file to the same folder as my index.html file (the penny finally dropped, sorry for being so lame, but this is all so new to me).
However when I tested it with the Google Analyse robots.txt - it still states my pages are 'Allowed'.
Where do you think I have gone wrong?
1. Select Insert > Add-ons > Publish File...
2. Click the Add button, and select the file you want to publish from your hard drive. (in this case your robots.txt file)
3. Click OK.

and then republish your site

taken from the help file available on their site
http://www.webpage-maker.com/files/wpm.pdf
Question Author
ChuckFickens: I have done as you instructed and the file robots.txt (appears as: robots.txt.txt on the WS-FTP) is positioned below the index.html file.
Unforntunately when I check it against the Google Analyse robots.txt - it still states my pages are 'Allowed!'
Question Author
ChuckFickens: My robots.txt file is up and running at last! I had made a mistake in the text file.
Thank you so much for your help x
-- answer removed --

1 to 10 of 10rss feed

Related Questions

Sorry, we can't find any related questions. Try using the search bar at the top of the page to search for some keywords, or choose a topic and submit your own question.