Get a List of All the URLs from a Website

Find and create a list of all the urls of a particular website

You might need to do this if you’re moving to a new permalink structure and need to 301 redirect the pages. For large sites, a lot of time can be saved by making good use of free sitemap generators online and excel.

A bunch of the online sitemap generator tools either ask for your email or have a maximum number of pages that they index. This one: www.xsitemap.com is my favorite since it works well, doesn’t ask for an email, and as far as I can tell, has no limit on the number of pages. I just ran it for a site with 2500 pages. It took about a half hour.

get-list-of-all-urls-website

After it’s finished, you have two options for grabbing the data. Either you can copy and paste from the readout or download the file provided.

Once you’ve pasted the data into excel, use the find/replace function to eliminate all the data you don’t need. For 301 redirects, you can get rid of anything extraneous, which means everything prior to the suffix, including the root domain.

301 redirect structure looks like this:

redirect 301 /old/old.htm http://www.you.com/new.htm

14 replies
  1. Peter Donalds
    Peter Donalds says:

    This is a really interesting strategy, although I am still trying to get my head around it, to understand it better.

    Reply
  2. Raheel Farooq
    Raheel Farooq says:

    Thank you, Ryan. I needed the list of my website URLs for SEO purposes. Your trick worked.
    Have a nice day!

    Reply
  3. baadshah
    baadshah says:

    looking for advance version of this feature.
    I am looking for filter on page show only pages with extension which i need
    also i want to view links upto certain level, basically i can give criteria to exclude some of sub links.

    Reply
  4. Ajit
    Ajit says:

    I have a website which is password protected. When I enter URL as http: //Loginname:Password@URL it gives only 23 pages where in reality there are more than 3000 pages.

    What is the solution for this problem?

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply to Simon G Cancel reply

Your email address will not be published. Required fields are marked *