#Anomaly 2 demo update#
#Anomaly 2 demo code#
#Anomaly 2 demo series#

#Anomaly 2 demo series#

using VALUES is much nicer.Īfter generating the axis, we left join the actual data into it to get a complete series for each status code. In the old days, to generate data we had to use a bunch of SELECT. You might be familiar with the VALUES clause from INSERT statements.

VALUES list: special clause that can generate 'constant tables', as the documentation calls it.

generate_series: function that generates a range of values.

To generate the axis we used two nice features of PostgreSQL: status_code ) GROUP BY period, status_code įirst we generate an axis using a cartesian join between the status codes we want to track, and the times we want to monitor. status_code, count ( * ) AS entries FROM axis a LEFT JOIN server_log l ON ( date_trunc ( 'minute', l. Correct! WITH axis AS ( SELECT status_code, generate_series ( date_trunc ( 'minute', now ()), date_trunc ( 'minute', now () - interval '1 hour' ), interval '1 minute' * - 1 ) AS period FROM ( VALUES ( 200 ), ( 400 ), ( 404 ), ( 500 ) ) AS t ( status_code ) ) SELECT a. Parsing and processing logs is outside the scope of this article, so let's assume we did that and we have a table that looks like this: Either way, you probably want to check where it's coming from.

A sudden increase in 200 status code: You either have some significant legit traffic coming in, or you are under a DOS attack.

Did you move some pages and forgot to set up redirects? Is there some script kiddy running a scan on your site?

A sudden increase in 404 status code: You may have an SEO problem.

#Anomaly 2 demo update#

Did you change some validation logic and forgot to update the client? Did you make a change and forgot to handle backward compatibility? A sudden increase in 400 status code: You may have a problem in the client.Did you just push a new version? Is there an external service you're using that started failing in unexpected ways? A sudden increase in 500 status code: You may have a problem in the server.

To illustrate the type of insight we can draw from just this data:

#Anomaly 2 demo code#

We are going to analyze logs of a web application, so the data we are most interested in is the timestamp and the status code of every response from the server. The data in these logs can be extremely useful in identifying anomalies. Later we'll see how using backtesting can help us identify ideal values.Īpplication servers such as nginx, Apache and IIS write a lot of useful information to access logs. The quality of our results are directly related to the parameters we set for the query. When we increased the z-score threshold to 3 we did not identify any anomaly.

When we decreased the z-score threshold to 0.5, we identified the value 2 as an anomaly in addition to the value 12.For example, let's see what anomalies we identify when the z-score is greater than 0.5 and when it's greater than 3: Changing the z-score threshold can affect our results. So far we used one standard deviation from the mean, or a z-score of ☑ to identify anomalies. Using z-score, we also identified 12 as an anomaly in this series.