Web Log Analysis

This is in my opinion one of the most over looked issues of running a web site. Not only can you learn about things like which search engine keywords are being used to find your site, and which search engines are spidering your site, you also increase the security of your site because you can pick up on hack attempts and lock the offenders out of your server. You can also often eliminate a huge amount of traffic by blocking offender, which will free up bandwidth and your CPU for valid customers. On one of my sites I found that over 80% of the traffic was bogus... Either bad bots, spidering my site, attempting to hack my site, or collecting email addresses or other information.

Things to look for:

404 Errors:

This is the first thing that I look at. It can tell you if you have any bad links on your site, and can also show people or bots snooping around your site or trying to find programs that they can abuse.

500 Errors:

These are script errors and this can let you know if there is a problem with a script on your site, and can sometimes be a sign of someone trying to hack a script.

401 Errors:

If you have a passworded area on your site then this will tell you about failed login attempts. Good to keep an eye on to see if someone is attempting to hack into the passworded area. Also keep an eye out for people logging in at the same time from different IP addresses.

403 Errors:

Once you start to put filters in place this will allow you to see who you are blocking and you can make sure that you have not made any mistakes.

Search Engine Keywords:

Like I mentioned previously you can extract a list of what keywords and search engines have been used to access your site.

Linked Sites:

You can find out from the referrer value which sites are linking to you. It is not extremely accurate now a days because many people are running firewall software that strips out the referrer value so you don't really know where they came from

File Downloads:

If you offer file downloads then you can track how many times files were downloaded.

Hits Per Page:

Figure out which pages are the most popular with your visitors.


You can also track the number of unique visitors to your site and also get an idea of the average number of page views per visitor.

These are just a few of the things that I look at from my web sites on a daily basis. Of course I use a program to parse my server logs into a readable report that I can then glance through to see what is going on. I HIGHLY recommend this for any one serious about there web site.