In the first part of this series of posts on analysing web analytics data, I covered how the initial stage in understanding the performance of a website during the previous week is to compare the numbers for that week against a set of reference numbers. While there are various alternatives to what reference numbers to use, the ideal option is a forecast. This takes into account historical performance and all known factors that would impact on performance. This post covers the typical next step when an issue with performance has been discovered.
Trending the data and looking for patterns
Once a difference in the data has been discovered (positive or negative), my typical next step is to trend the data over a longer date range, keeping it at a weekly level. If you were comparing performance against an alternative set of numbers (as opposed to just the previous week), then trend the comparison numbers too, whether they are from last year, a benchmark, a target or a forecast.
There are undoubtedly more scientific ways of doing this but I generally look for patterns in the data visually, either in the web analytics tool or by charting in Excel. An alternative is to trend the % difference between the actual performance and the comparison number, again looking for when this % difference changes. There are generally four options for what you might observe in the data:
- there is a relatively consistent difference over time
- the numbers are quite erratic with no clear pattern
- the difference is gradually increasing over time
- the difference has only recently appeared
Consistent difference over time
It can be the case that you are using comparison numbers to understand performance and when you trend both sets of numbers, you notice that there is always a difference between the two. It got flagged up to you this week as the difference was a little bigger than normal.
My advice here is to choose a different set of comparison numbers or to normalise the comparison set. Make sure that the numbers you are looking at are useful to you, if they are not, change them. This may mean revising the forecast or knowing that performance is always 15% to 20% up on the previous year.
The numbers are erratic
This one is a little harder to deal with and more information is required. If the number of people impacting on this metric is low, it might simply be that it is difficult to achieve a consistent average performance on a regular basis. Alternatively, if traffic numbers are high, then there is something else affecting the behaviour of website visitors. In this case, segmentation is your best friend.
An example of this would be the website conversion rate fluctuating significantly over time. Segmenting by traffic source then demonstrates that the conversion rate for each traffic source is quite steady although it varies a lot between traffic sources. The fluctuation in overall conversion rate is actually caused by email bursts with email having a significantly lower conversion rate to the other traffic sources. Whenever an email is sent out, the proportion of traffic from email goes from 3% to 23% with the average website conversion rate dropping in line with the lower quality traffic.
Difference increasing over time
Sometimes the numbers start moving in one direction or another. It is difficult to pinpoint exactly when this trend started and as the difference is increasing over time, it definitely needs to be understood. This is another case when you would look to segment the data.
It might be similar to the example from Erratic Numbers, except that in this case, the traffic source mix is changing over time. An example for this would be an SEO strategy kicking in with more traffic driven by SEO to the article pages. These pages have a much higher than average bounce rate as visitors are only interested in reading about a single article leading to the Bounce Rate for the website trending up.
The website is still performing at the exactly same level, just attracting traffic of a difference level. Changes are not necessarily required, more of an education process internally to explain the increase in the Bounce Rate. The focus might be better shifted from the website Bounce Rate to the Bounce Rate for e-commerce pages or the incremental sales through these visitors attracted to the site by the articles.
If there appears to be a recent change in performance, you should look at the data at a daily level, for at least the past two and preferably four weeks.
As with the weekly data, you are looking for patterns in the data, which generally appears as one of three options:
There was a change for one or more days but the numbers have already reverted to a more typical level. While the weekly performance looked wrong for this week, as it was a temporary spike you know that they will look normal next week. Ideally you identify what caused the change but it is not worth investing a large amount of time on it as the situation has already been resolved. If it reoccurs or if it was a positive change that you wish to replicate, then it would definitely be worth investigating.
On a certain day, the numbers changed and unlike the spike, they haven’t reverted back. The trick now is to identify what changed to the website, to marketing or for the business to cause this step change to occur. If it helps, go a step further and pinpoint the hour in which the change occurred – it’s always fun to call the marketing or web dev team and ask them what they did between 3pm and 4pm the previous Thursday. Whether it is the day or the hour, knowing the time of the change gives an insight into what factor has caused the change and this starts the process of deciding what actions can be taken if required.
Identifying a step change can prove very useful for evaluating the business impact of a change or for forecasting the business impact of proposed future changes. It is fairly easy to see what the numbers approximately would have been if the change had not been made and so the % difference can be attributed to that change, not as good as an A/B test but still very insightful.
The situation here is the same as with the weekly trend, the numbers have starting trending up or down without a clear start point. As such, the method used to approach this pattern in the data is the same as for the weekly trend pattern.
This post has covered another vital technique in analysing web analytics data, trending data and looking for patterns. Note that while the charts used as examples here appear to be for visits or maybe orders, the same principle can and should be applied to metrics like bounce rate or page views per visit. I actually should have reread this post myself during the week when an increase in the Bounce Rate for a client last week was eventually exposed as a temporary spike upon trending that metric at a daily level.
Next time (and hopefully it won’t be a three week gap this time), I will move onto segmentation and some examples of situations where segmentation can prove to be useful in understanding performance, both what numbers to apply the segments to and some segments that I have found useful.
Links to other posts in the series