LightHouse
Silver Contributor
FASTLANE INSIDER
EPIC CONTRIBUTOR
LEGACY MEMBER
Read Fastlane!
Read Unscripted!
SiteMAPS 101
Wiki sez:
ALL SITEMAPS are NOT created equal. Case in point: 1and1
1and1 has a slick feature in the control panel to automatically create a sitemap for you and you can submit it once a day. Well decided to check it out on an old site I used it for today.
It had everything I DID NOT want indexed in there (/admin, /temp, etc) and not one product or information page. Webmaster tools completely rejected it and it had been on my site for a year. This does NOT help my indexing of products thus not giving all my products a shot at ranking.
So you need a fix? Here is how to create one that is useful and where the two most important places to submit it are.
Go to Google Sitemap Generator for Windows :: GSiteCrawler and download the program FOR FREE. Gsitecrawler will crawl your site from the inside out and get every URL there is to get.
The advantage to this tool is two fold.
#1 you have a ban URL and exclusion feature. This is great for lists with multiple pages print icons and everything else that adds excess URL’s that are not landing page ready. The nice thing is if you see a line like ?print=1 URL’s coming up you can pause the crawler (required) add it into the exclusions and hit re-filter, it will remove those and then you can continue until you only have first level product pages.
#2 is that you get a wealth of info and you can export a sitemap in text or XML, a robots.txt, and a urllist.txt for yahoo.
You can also set the program to upload via FTP so when you add new products bring the project back up. Re-crawl the site and everything is done on auto pilot.
What do I do now that I have a great sitemap?
Upload both files you exported after you verified you have all the solid URL’s you want in the file. Upload them to your root directory. You can also load your robots.txt so the robots will not index anything you do not want them to see.
Google: Go to webmaster tools (https://www.google.com/webmasters/tools/) If you have been on the internet in the past few years chances are you have already seen this for your site. Go to the sitemaps area and show Google where the sitemap is on your server.
(If not create and account and go through the verification process, it’s a great way to see how Google see’s your site as well as you can use the optimizer when you want to A/B test in the future. )
Yahoo: Go to yahoo’s site explorer (https://siteexplorer.search.yahoo.com/mysites) Sign up and verify your sites as well. These will provide good data later. Add your site then select it, it will bring up a sub menu for feeds. Add the link to your urllist.txt to the box and select data feed and click ok!
MSN:
WHAT did I do all this for?
Well you helped the search engines index you’re most valuable pages. Another great thing to do is link it in the footer of your homepage, but this is not required. It helps for other indexing sites to find the same information.
Time spent: 20 mins hands on and depending on how large your site is 30 seconds to 1 hour for g-site crawler to run the URL’s
-Chris Waldron
Wiki sez:
Do you have one??? Do not rely on the one your hosting package creates to be good.
ALL SITEMAPS are NOT created equal. Case in point: 1and1
1and1 has a slick feature in the control panel to automatically create a sitemap for you and you can submit it once a day. Well decided to check it out on an old site I used it for today.
It had everything I DID NOT want indexed in there (/admin, /temp, etc) and not one product or information page. Webmaster tools completely rejected it and it had been on my site for a year. This does NOT help my indexing of products thus not giving all my products a shot at ranking.
So you need a fix? Here is how to create one that is useful and where the two most important places to submit it are.
Go to Google Sitemap Generator for Windows :: GSiteCrawler and download the program FOR FREE. Gsitecrawler will crawl your site from the inside out and get every URL there is to get.
The advantage to this tool is two fold.
#1 you have a ban URL and exclusion feature. This is great for lists with multiple pages print icons and everything else that adds excess URL’s that are not landing page ready. The nice thing is if you see a line like ?print=1 URL’s coming up you can pause the crawler (required) add it into the exclusions and hit re-filter, it will remove those and then you can continue until you only have first level product pages.
#2 is that you get a wealth of info and you can export a sitemap in text or XML, a robots.txt, and a urllist.txt for yahoo.
You can also set the program to upload via FTP so when you add new products bring the project back up. Re-crawl the site and everything is done on auto pilot.
What do I do now that I have a great sitemap?
Upload both files you exported after you verified you have all the solid URL’s you want in the file. Upload them to your root directory. You can also load your robots.txt so the robots will not index anything you do not want them to see.
Google: Go to webmaster tools (https://www.google.com/webmasters/tools/) If you have been on the internet in the past few years chances are you have already seen this for your site. Go to the sitemaps area and show Google where the sitemap is on your server.
(If not create and account and go through the verification process, it’s a great way to see how Google see’s your site as well as you can use the optimizer when you want to A/B test in the future. )
Yahoo: Go to yahoo’s site explorer (https://siteexplorer.search.yahoo.com/mysites) Sign up and verify your sites as well. These will provide good data later. Add your site then select it, it will bring up a sub menu for feeds. Add the link to your urllist.txt to the box and select data feed and click ok!
MSN:
WHAT did I do all this for?
Well you helped the search engines index you’re most valuable pages. Another great thing to do is link it in the footer of your homepage, but this is not required. It helps for other indexing sites to find the same information.
Time spent: 20 mins hands on and depending on how large your site is 30 seconds to 1 hour for g-site crawler to run the URL’s
-Chris Waldron
Dislike ads? Become a Fastlane member:
Subscribe today and surround yourself with winners and millionaire mentors, not those broke friends who only want to drink beer and play video games. :-)
Last edited:
Membership Required: Upgrade to Expose Nearly 1,000,000 Posts
Ready to Unleash the Millionaire Entrepreneur in You?
Become a member of the Fastlane Forum, the private community founded by best-selling author and multi-millionaire entrepreneur MJ DeMarco. Since 2007, MJ DeMarco has poured his heart and soul into the Fastlane Forum, helping entrepreneurs reclaim their time, win their financial freedom, and live their best life.
With more than 39,000 posts packed with insights, strategies, and advice, you’re not just a member—you’re stepping into MJ’s inner-circle, a place where you’ll never be left alone.
Become a member and gain immediate access to...
- Active Community: Ever join a community only to find it DEAD? Not at Fastlane! As you can see from our home page, life-changing content is posted dozens of times daily.
- Exclusive Insights: Direct access to MJ DeMarco’s daily contributions and wisdom.
- Powerful Networking Opportunities: Connect with a diverse group of successful entrepreneurs who can offer mentorship, collaboration, and opportunities.
- Proven Strategies: Learn from the best in the business, with actionable advice and strategies that can accelerate your success.
"You are the average of the five people you surround yourself with the most..."
Who are you surrounding yourself with? Surround yourself with millionaire success. Join Fastlane today!
Join Today
