Using the example of a well-known source of predictions for football matches called FiveThirtyEight in the world, we will try to assess their accuracy and compare them with the odds of the bookmaker’s office Pinnacle. Read on to find out if they have any value for bettors.

Basically, the bookmaker’s odds are a forward-looking prediction of the likelihood of different outcomes for a sporting event. They are most often expressed in decimal notation, and then you just need to invert the numbers to get the percentage of the probability of the corresponding outcome. That being said, it makes sense to do a bit of work to eliminate the bookmaker’s margin. A factor of 2.00, for example (in this case, the margin has already been deducted), assume a probability of an outcome equal to 1 / 2.00 – 0.5, or 50%.

Of course, the bookmaker can be wrong too. The mistakes of the bookmakers give their clients the opportunity to find the expected value and make a profit in the long term. Readers of this article, however, should already be aware that the mistakes of the Pinnacle bookmaker, including those for the football betting market, are randomly distributed, and in the general picture of affairs the odds of this bookmaker fairly reliably reflect the actual odds on the outcome of events.

From this point of view, the best choice is the closing line odds, which, among other things, can be used to predict how much profit a bettor can expect.

However, there are numerous other groups of forecasters offering their own predictions of the outcome of football matches. In our country, in recent years, they have been growing like mushrooms after rain, offering both paid and free forecasts for sports events. At the same time, paid predictions “guarantee”, according to their assurances, 70% and even 80% cross-country ability. Figures that even a betting guru can only dream of.

One of such well-known groups of tipsters in the world is the Fivethirtyeight.com platform. It publishes forecasts of political, economic and sports events. It was originally a blog created by Nate Silver, an American statistician, former poker player and author of Signal and Noise. He made a name for himself by successfully predicting the outcome of the US presidential election for 49 of the 50 states in 2008, and four years later he was able to predict the outcome for all 50 states. The result, you see, is impressive.

For the sports fan or bettor, FiveThirtyEight’s predictions on match outcomes are especially helpful as they provide specific odds for home wins, away wins and draws. By inverting these values, we can instantly figure out the estimated fair ratios.

The goal of any bettor looking for valuable bets is to find fair odds that more accurately reflect the probability of an outcome than the bookmaker’s odds. If the player has such an opportunity, then he just needs to place bets with the bookmaker’s odds at the moment in time when they have a higher value. And if a player manages to make a profit in the long term, then this is a sure sign of his ability to calculate more accurate odds than the bookmaker suggests. Let’s try to understand how FiveThirtyEight lives up to the aspirations of its subscribers.

** FiveThirtyEight Football Predictive Model **

FiveThirtyEight Tipsters first published their soccer predictions in January 2017, although their database of results contains entries dated August 2016. The methodology of the authors of this blog is based on the “substantially revised version of ESPN’s Football Team Strength Index (SPI)” originally developed by Nate Silver. This model uses the expected goals indicator (or xG) and the results of the analysis of the statistical Poisson distribution, based on which a matrix of possible match outcomes is generated, which allows calculating the probabilities of a home win, an away win and a draw.

FiveThirtyEight forecasters believe that their predictions for football matches are very good, and their recommendations are much more valuable than trying to guess the result without using skills. Could their predictions really be more accurate than the assumed probabilities from the Pinnacle company? Let’s try to figure it out honestly and impartially.

** Are FiveThirtyEight’s predictions profitable? **

After analyzing the database with the history of the closing odds of the Pinnacle bookmaker, as well as considering the alleged probabilities of the FiveThirtyEight platform open for research, we have compiled a sample of more than 16 thousand matches of various European football leagues played between August 12, 2016 and March 31, 2019 and compared almost 50 thousand pairs of odds for home win, away win and draw.

In 20,093 cases, Pinnacle’s closing odds were higher than FiveThirtyEight’s implied probabilities. The average by which these odds (with an average of 4.12) were higher in comparison was 16.2%, which means that if we were willing to place bets of the same size with the offered odds, we would be able to make a profit of 16 , 2%, assuming, on average, FiveThirtyEight odds are an accurate or efficient representation of “fair” odds. In fact, these odds showed a -6.0% loss, even worse than a -4.3% loss for bets placed with all 49,905 odds.

The Chart 1 below shows how FiveThirtyEight’s implied odds have failed to predict actual betting returns with Pinnacle’s closing odds. If we divide Pinnacle’s closing line odds by the estimated FiveThirtyEight odds, we get the expected profit for the selected bet (assuming the FiveThirtyEight odds are effective).

Now, let’s group the bets according to the incremental increase in the expected profit (in increments of 0.01): you can see that the expected profit does not correlate with the actual profit from the bets at all. Regardless of the ratio of the closing line odds of the Pinnacle bookmaker to the estimated FiveThirtyEight odds, the average profit remains negative at -6%. It turns out that FiveThirtyEight’s odds do not contain predictive value at all compared to the closing odds of the Pinnacle bookmaker.

But what will change if we look at the situation from the opposite side? This time, let’s assume that FiveThirtyEight acts as the bookmaker, and Pinnacle offers only a predictive model. To measure the expected profit, we will use the ratio of the FiveThirtyEight odds to the fair odds of the Pinnacle closing line minus the margin.

Betting with FiveThirtyEight odds for 25,557 times that exceed the fair closing odds at Pinnacle offers an actual return of 15.5%, which is very close to the 15.9% average advantage (average odds equal to 4.49). Chart 2 shows a strong correlation between expected and actual returns for this inverted hypothesis. The slope of the trend line is almost unique and passes through the origin (shown here is the equation y = mx + c). This indicates that Pinnacle’s closing odds are highly effective, not the reputable platform.

** Are the comparisons correct? **

When publishing these studies on Twitter, reads pointed out that we are, in fact, comparing apples to oranges. FiveThirtyEight’s predictive probabilities are calculated before the start of the competition, and the final probability score is published even before the team’s penultimate match ends. More than one day will pass before the match in question. And the degree of accuracy of the published forecasts entirely depends on the information available at the time of their formation.

At the same time, Pinnacle’s betting line closing odds reflect all the information available on the market before the start of the match. These odds will take into account factors such as injuries to players, changes in the composition of the teams, weather conditions and the state of the field – everything that the FiveThirtyEight platform cannot take into account for objective reasons.

To make a completely honest comparison of the predictive model with the closing odds of the Pinnacle bookmaker, it would be necessary to convince the tipsters from FiveThirtyEight to calculate and publish the predictions at the same time with the Pinnacle bookmaker, that is, right before the start of the match. But this, of course, will not happen. Alternatively, we could use the Pinnacle betting odds published at the same time as FiveThirtyEight’s predictions for the respective match. Alas, we do not have data on the odds of the bookmaker’s office Pinnacle with timestamps, and even if they could be found, the published odds of the opening line of the bookmaker’s office Pinnacle still appear later than the final publications of FiveThirtyEight’s forecasts.

But even with all of the above, using Pinnacle’s opening line odds has the potential to offer a more honest comparison of patterns than using the closing odds. Let’s take a look at the results. In 18,952 cases, Pinnacle’s opening line odds were higher than FiveThirtyEight’s odds (average: 3.97), with an average advantage of 14.2%. These cases were matched by a –4.1% loss, slightly better than a –4.4% loss when placing bets at all 49,905 odds. As before, here we see almost no correlation between expected and actual profit, which is reflected in Chart 3 .

Repeating the comparison of the models in the inversion, and using the opening prices of the Pinnacle book office as “true”, we will see that the level of correlation will be much higher – not as accurate as for the closing prices, but still close to equality. Placing bets with FiveThirtyEight odds for 25,775 cases in which they outperform Pinnacle’s fair odds offer an actual return of 12.8%, again relatively close to the average advantage of 14.8% (averaged 4.54 ).

** Signal and Noise **

The data we studied speaks for itself. In our opinion, the predictive capabilities of FiveThirtyEight from the point of view of information content are inferior to the coefficients of the Pinnacle bookmaker, both due to the time difference between the publication and the reason that the correct formation of odds is the basis of the Pinnacle bookmaker’s business model, and the FiveThirtyEight blog exists more for players betting for fun. FiveThirtyEight tipsters’ income does not depend on their forecasts, or at least does not directly depend on forecasts. In fairness, it is also worth mentioning that the authors of FiveThirtyEight do not set out to shape their predictions in such a way that they are suitable for use in betting.

Let’s try another interesting thought experiment. By combining both predictive models, you can try to create a third model, the effectiveness of which will exceed the individual coefficients of the Pinnacle bookmaker. This way we can confirm or deny that FiveThirtyEight’s predictions contain some useful data that could complement the useful data contained in the Pinnacle bookmaker’s odds.

So far, our analysis boiled down to choosing one of two options: which of the two predictive models is the more accurate source of information. The responses received were both comprehensive and predictable. Let’s add some intrigue and reformulate our research question.

Let’s say the final probability of any considered outcome = Z * (the probability of this outcome occurring according to the FiveThirtyEight version) + (1 – Z) * (the probability of this outcome occurring according to the Pinnacle bookmaker’s version) for each Z, the value of which corresponds to the condition 0 ≤ Z ≤ 1. Let’s see what value of Z will maximize the predictive value of the final probabilities.

Such a model will be flexible enough to be able to calculate finite probabilities based on predictions owned exclusively by FiveThirtyEight (if we take Z = 1), exclusively by Pinnacle BC (if Z = 0), or based on any intermediate option (if 0 ≤ Z ≤ 1).

But how to determine the optimal Z value? There are several possible options, but we will use maximum likelihood estimation (MLE). The goal of MLE is to find the values of one or more unknown parameters that best “fit” the set of data under study. How do we measure the degree to which a found value is “fit”? This problem is solved by comparing the likelihood of the observed values depending on the value of the unknown parameter and what we have already observed before.

The model we have formed above has a single parameter: Z. Any of the values of Z will allow us to calculate for each match in this dataset a set of “final” probabilities of a home win, a draw or an away win, depending on this particular Z value. For each In the match, the likelihood of the observed values, in accordance with what we have already observed, will be our final probability of a home win, a draw, or an away win if the match results in a home win, a tie or an away win, respectively.

For example, consider the situation for the probabilities of a home win, a draw result and an away win, respectively equal to 0.5, 0.3 and 0.2. In other words, if the match results in a tie, then the likelihood of the observed data for what we have already observed is 0.3.

Since individual matches are independent events, the likelihood for a set of outcomes that is identical to previously observed outcomes will be the product of the likelihoods for each individual match. Maximizing this product will be the goal for which we will try to adjust the Z value.

It is clear that the product of 16 635 probabilities is infinitely small (you can try to imagine a bet of 16 635 bets). Thus, we face the problem of calculating the exact value when calculating the MLE using tools such as Excel: software restrictions do not allow calculating small values below a certain threshold, because everything that is less than this limit is rounded to zero.

To get around this problem, we can try to maximize the log likelihood. Since the absolute value of the degree of likelihood is irrelevant for the procedure for calculating the MLE (the only thing that matters to us is the nature of the change in the degree of likelihood depending on the adjustment of the parameters), maximizing the logarithm of the degree of likelihood will be mathematically equivalent to the operation we need.

Besides the fact that we will take the logarithms of the final probabilities of the outcomes of the match, instead of the product, we will calculate the sum of the logarithms of the degrees of likelihood. Performing this analytical procedure on an existing dataset will give us the following results:

- When using the Pinnacle BC close line odds, the log likelihood reaches its maximum at Z = 0. In other words, FiveThirtyEight forecasts do not add any optimizing effect to the final forecasting model when only FiveThirtyEight and Pinnacle BC are considered.
- When using Pinnacle Opening odds, the log likelihood reaches a maximum at Z = 0.04. Thus, the efficiency of the final model is about 4% dependent on the FiveThirtyEight data.

** Collective Odds Models **

The second result is the most interesting for us. If you need to choose only one data source, then the Pinnacle bookmaker’s opening line has a predictive value that objectively significantly exceeds the value of FiveThirtyEight’s forecasts. However, there is one thing worth pondering: the weighted average of a combination of 4% of FiveThirtyEight’s forecasts and 96% of Pinnacle’s opening line odds has a higher predictive value than forecasts of either side individually.

But is there any value to this result for bettors? On a practical level, it is not that significant. 4% is too small to be applied and, in addition, it can be of little statistical nature. But what if we chose a higher Z value? And how would things change if instead of two predictions we had a much larger number of predictions, each with its own Z value? We are faced with a variation of the “collective opinion” theory, according to which a combination of individual predictions may be more valuable than one single prediction with the highest value.

In fact, it is thanks to the action of this theory that the Pinnacle bookmaker’s odds are so accurate. The line values are set by the most informed and professional traders. In addition, they allow professional players to enter the game, rather than deny them participation – such players act as an auxiliary source of information for adjusting the values of the lines, making the odds even more accurate. Pinnacle’s closing line odds are essentially “collective predictive models” and reflect finite probabilities. Therefore, the quality of FiveThirtyEight’s forecasts will always be objectively lower.