botify review: crawl & server log analyzer tool
while operating as an agency, both in-house or freelance consultant, having the right tools for the job can make your life easier and your campaigns more effective. as seo has become a staple of the digital marketing mix, we see more tools released onto the market that range from free to a few hundred dollars per month or more. with so many to choose from, it’s hard to know which will provide value in the long term on your investment.
one area of seo i have concentrated on in the past few months is analyzing server logs for some of my clients. i’ve tested a few different server log tools but none of which were specifically targeted to seo needs. i have even played with grep but that process can become cumbersome when dealing with gigabytes of data. in my quest to find the perfect tool, i came across botify.
in this review, i will talk about my experience using botify, a crawling and server log analyzing tool that helps site owners answer, “what do you really know about your website?“.
crawl optimization
ratio of urls found by googlebot and botify
botify compares the number of urls found by its crawl of the site and googlebot’s crawl as reported by server logs.
botify splits their tool into two separate features:
analyzing issues hindering site crawling
analyzing googlebot’s crawl patterns
issues hindering site crawling are found through crawls made by their own bot. googlebot’s crawl patterns are analyzed through the supplied server logs. while you can opt to use one or the other, subscribing to both services will combine the crawl and server log data into a single report for additional insight into crawling patterns and issues.
that said, i would argue their server log analysis tool is by far their most valuable offering, especially for large web sites. it’s a rarely seen feature in seo tools today but one i have found to offer a real glimpse into how googlebot (and other bots, too) are prioritizing your content.
getting started with log analyzer
seo visits overview
botify reports an seo visit as one or more visits to a page from an organic source as reported by server logs.
getting started is pretty simple. i had to upload the server logs to their server and botify would take care of the rest. they recommended having at least a month’s worth of data imported to have an accurate view of server activity. once everything is configured and you have access to the dashboard, the real fun starts.
you can segment the data by three different data points:
dimension type, such as page type, host, or mimetype
time frame the data was captured as a pre-configured range or custom date ranges
bot type, such as googlebot images, google smartphone, or bing
moving around the dashboard, you will find various reports with details around sets of pages or specific urls within a set of pages.
unique urls crawled by tag
botify groups pages by their url structure, known as tags, and reports over time the activity recorded such as the amount of unique urls crawled.
as you drill down into the data sets, you can start see over time the distribution of all this data. it gives you a great idea as to what the short and long term trends of your site are in relation to bot crawling. once you have the data you need, botify makes it easy to extract with the option to export as a csv.
crawl & log reports
some reports botify includes but not limited to are:
ratio of missing, unique, and duplicate html tags (i.e., titles, meta descriptions, and h1s)
distribution of http codes over time by bot
number of unique crawls vs number of visits by tag
average internal links per tag
the above are pretty standard reports one would expect from a crawling tool. however, while i can’t touch upon everything today, i will go through a few reports that really stood out to me while using botify.
crawled/uncrawled pages by level of depth
screaming frog and other tools will tell you the distribution of pages at each level of depth but did you know how many of those pages by each level of depth are actually crawled by google? botify knows and while it’s at it, botify also presents the number of urls by depth that also had at least one visit (according to the user-agent string).
ratio of crawled and uncrawled pages
crawled and uncrawled pages by level of depth by googlebot.
if you’re not interested seeing it by level of depth, you can easily adjust the report and view the same data by category to get a different perspective on how much crawling each section of your site is receiving.
daily volume of crawling
curious to see the first time a page was crawled by google or when the last time it was crawled before a visit occurred? with the date picker, you can compare different time frames in order to understand when crawling is increasing or decreasing over time (it’s been hinted at that crawl volume may impact how often a page receives traffic but of course you will need to dig into your own data to find out.)
crawl retention
crawl timeframe by googlebot before a visit was recorded.
average delay distribution
average delay distribution
the more time googlebot takes to download a page’s content means less time crawling deeper into your site.
botify also includes a load time report. using their own crawler, botify records by category which sets of pages take less than 1 or more than 2 seconds to load. it’s not as in depth as other load time tools are but in bulk it provides a great view of how quickly your site loads across site categories.
orphan pages
orphan pages
botify reports on the distribution of orphan pages crawled by google
orphan pages are pages found in the server logs but not in the structure of the site. in other words, these pages are not linked from within the structure identified by botify. however, this particular report displays all orphan pages not linked within the structure but still crawled by google.
pricing
as mentioned, their crawler and log analyzer are two separate platforms and therefore both have different pricing structures. the pricing options for the crawler is a bit overwhelming with monthly and pay as you go tiers, but i believe botify does a good job catering to those not looking for much or not knowing how much they need. small agencies can expect to pay anywhere between $50-$300 a month while large agencies will most likely opt for payments at around $400-$600 a month.
the log analyzer is the most expensive option out of the two. you really need to justify this cost and have an immediate plan of action. for large sites, you will want as much data as possible analyzed for the log reports to be valuable. expect to pay $1,000-$1,500 per month with this tool.
the verdict
for me, botify is the best log analyzer out there right now for enterpise seos. it comes at a time when sites are at their largest and site owners are having a hard time leveraging all of this data in a meaningful way. the features covered in this review are only scratching the surface and when combined with other reports opens up the possibility for interesting analysis.
while i was pleased with the entire experience, i do want to mention a few things botify can improve upon in the future:
add support for different bots, such as baidu or even facebook
add a glossary for all of the terms used in the reports
add tips to why this data is useful/how it can be used
create a more seamless experience when accessing the log and crawler dashboards
if you have any feedback of your own based your experiences with botify, sound off in the comments and let me know what you think.