Random Quote

Guys are simple… women are not simple and they always assume that men must be just as complicated as they are, only way more mysterious. The whole point is guys are not thinking much. They are just what they appear to be. Tragically. — Dave Barry


Enter your email address below to receive notifications of new posts.

Investor sentiment for the S&P 500

One of the advantages of social media analytics is the ability to drill in to the investor sentiment about a particular company or instrument. But can we say something useful about the aggregate stock market based on the social media traffic about individual companies? And does the aggregate sentiment merely reflect the market performance, or is there an independent component? These are the kinds of questions we explore today, focusing on the social media sentiment for the U.S. stock market.

We’ll look at the sentiment for the S&P 500 index in this post. The analysis is based on PsychSignal data, where the history is readily available. The data does not include indicators for the aggregate stock market, but we’ll construct one from the stock index constituents.

It is always useful to have some idea of what the underlying data looks like, so let’s start by having a look at the distribution of the sentiment by stock in the sample period. Chart 1 shows a histogram of the average bullish and bearish sentiment by constituent from September 2009 (this is the effective inception month) through the middle of February 2015.

Chart 1. The empirical distribution of the average bullish and bearish social media sentiment for the constituent stocks in the S&P 500 index (with the sign of the bearish sentiment negated). The time period runs from September 2009 through the middle of February 2015. Sources: PsychSignal, own calculations.

It is evident that the social media sentiment has been predominantly positive in the last few years. This is no great wonder, given that we have been in a strong bull market for much of this time. It is also notable that there is a qualitative difference between the bullish and bearish sentiment readings (see Table 1). Both distributions are skewed but in the opposite ways, considering the directionality. It appears that if the investors are bullish about stocks, they hold those views strongly with only a scattering of mildly positive opinions. But if the investors are bearish, they often hold rather extreme negative views with less bulk in the main body of the distribution.

Table 1. Sample statistics for the average bullish and bearish social media sentiment for the constituent stocks in the S&P 500 index (with the sign of the bearish sentiment negated). The time period runs from September 2009 through the middle of February 2015. Sources: PsychSignal, own calculations.


If we look at the balance of the social media sentiment — i.e. the difference between the bullish and bearish readings — the distribution of the stock market sentiment loses its skew and the excess kurtosis (Chart 2).

Chart 2. The empirical distribution of the balance between the bullish and bearish social media sentiment for the constituent stocks in the S&P 500 index. The time period runs from September 2009 through the middle of February 2015. Sources: PsychSignal, own calculations.

In order to estimate the prevailing market sentiment, we’ll simply take the average of the bullish and bearish sentiment daily for the index constituents, and subtract one from the other. You could also use the ratio of the two, but you’d then run the risk of numerical problems at times when the bearish sentiment is very low. Looking at the balance between the bullish and bearish social media messages, this is what the stock market sentiment looks like for the S&P 500 index in recent years.

Chart 3. S&P 500 index (blue line) vs. the aggregated balance of bullish and bearish investor sentiment in the social media for the S&P 500 constituent stocks (orange line). The 30-day moving average of the sentiment is also shown (thick orange line). Sources: Quandl, PsychSignal.

It may be instructive just to let the eye rest on the above chart for a bit. In my mind, this is what the data suggests:

  • There’s quite a lot of noise in the social media sentiment.
  • There’s some amount of short-term mean reversion.
  • There are many occasions where the sentiment simply reflects the recent price action.
  • There may be instances where a rising or falling sentiment is a harbinger of market turning points.

It appears that the stock market index and the aggregate sentiment tend to move together: When the market goes up, sentiment improves, and vice versa. We can see this clearly in Chart 4, which shows a rolling 30-day stock index return vs. the 30-day change in the sentiment.

Chart 4. Rolling 30-day return for the S&P 500 index (blue line, left axis) vs. the 30-day difference in the social media sentiment (orange line, right axis). Sources: Quandl, PsychSignal, own calculations.

Just how long a memory do stock market investors have? In other words, how far into the future is the investor sentiment affected by equity performance? My sense is that this depends on many factors, including the poignancy and significance of historical market events. This would be a legitimate research question on its own, and we cannot aim for definitive answers here.

But for some insight, let’s calculate the correlation between the stock market return and sentiment changes over different time windows. The results (see Chart 5) indicate that the influence lasts a long time; it only starts to decay after a year, and gradually disappears some time in the third year. Interestingly, it is really only the bullish component of the investor sentiment that is significantly influenced by the market action. This is not what I’d have expected, but this is what the data are telling us.

Chart 5. The correlation between the S&P 500 return and changes in the social media sentiment as a function of the time window (the number of days). Sources: Quandl, PsychSignal, own calculations.

As to the mean reversion, Chart 6 shows the autocorrelation between daily changes in the bullish sentiment, and Chart 7 does the same for the bearish sentiment. It is indeed the case that there is significant negative autocorrelation from one day to the next1. This pattern is more pronounced in the case of the bearish sentiment.

Chart 6. The correlation of daily changes in the aggregated bullish investor sentiment in the social media for the S&P 500 constituent stocks. Sources: PsychSignal, own calculations.

Chart 7. The correlation of daily changes in the aggregated bearish investor sentiment in the social media for the S&P 500 constituent stocks. Sources: PsychSignal, own calculations.

As we’ve seen, there’s strong contemporaneous correlation between market moves and sentiment changes. To a trader or an investor, the real question is whether there’s useful residual information, i.e. whether the sentiment is useful for alpha generation or risk control purposes.

At first sight, it is difficult to see much evidence of predictability in the data. As an illustration, Chart 8 shows a scatter plot of the stock market sentiment vs. subsequent 5-day returns, sampled at non-overlapping 5-day intervals. All we can really say is that there’s quite a bit of noise in both the sentiment data and market prices. Were you to look at changes in the sentiment against the price action, you’d see a chart even noisier than this.

Chart 8. The aggregated social media sentiment (x-axis) vs. the subsequent return for the S&P 500 stock index (y-axis), sampled at non-overlapping 5-day intervals. Sources: Quandl,  PsychSignal, AAII, own calculations. 

But let’s take a step back. Market timing is notoriously difficult — many would say well nigh impossible — and it’s no surprise that there’s no clear-cut evidence of predictability in daily stock index levels. If there were, indeed, we’d probably have made a mistake somewhere along the way or somehow managed to look ahead. Nevertheless, we could certainly press on to identify and estimate a statistical model of market returns based on the sentiment inputs. I’ll spare you the torture, as I don’t think there much to be gained by such exercise.

Modern financial markets are mostly efficient, and it may well be that investor sentiment has little relevance for much of the time. However, it would be foolish to claim that the market is always efficient; there can still be occasions when the investor mood is important. Sometimes the investors get carried away too far in either direction, and I reckon it is good to be aware of this happening. Following this line of thought, perhaps there’s more information at the sentiment extremes. Let’s find out if this is so.

The specific question we are interested in is this: Is there a difference in the distribution of the near-term returns conditional on the sentiment level on those occasions when the sentiment is in the tail end of its distribution? In other words, what does the future look like if the sentiment today is either abnormally high or abnormally low?

Before we can answer the question, we need a way to identify the sentiment extremes. One way to do this is to standardise the smoothed sentiment, and then simply pick the days when the standardised measure exceeds a threshold. In this post, we apply a rolling Z-score to the rolling 5-day moving average of the social media sentiment, and use a signal threshold of ±2. Chart 9 shows the resulting signals, i.e. the days when the stock market sentiment strays into either tail2.

Chart 9. S&P 500 index (blue line) with the aggregated social media sentiment (orange line) and the days when the sentiment moves into abnormally high levels (green markers) or low levels (red markers). Sources: Quandl, PsychSignal, own calculations.

In order to put some numbers on this, we’ll next track the stock index returns from the close of the day after the entry signal (i.e. we ignore the calendar day when the signal is detected to avoid any look-ahead bias). Looking first at the bearish side, Chart 10 shows what the stock market performance looks like after sentiment troughs. As we see, the subsequent price action varies widely: Sometimes the market bounces back strongly, whilst other times it stays in the doldrums. On the average, though, it looks like the sentiment lows are good times to buy.

Chart 10. The average cumulative return for the S&P 500 index conditional on abnormally low prior social media sentiment (red line) vs. the unconditional return (blue line). The cumulative return after each of the occasions when the sentiment is low are also shown (grey lines). Sources: Quandl, PsychSignal, own calculations.

But hey, you will say: This was a bull market, so the prices probably went up regardless of when you bought into it. And this is indeed true. The market went up for most of the time in this time period, it just went up more — and significantly more — after the sentiment lows. If you peruse Chart 10 again, it also shows the average unconditional cumulative return. After a typical underperformance over a week or so, the market raced ahead at something like double the normal rate over the subsequent 15-20 trading days.

The story is different after the extreme highs of investor sentiment. There is no consistent tendency for either strong or weak stock prices after an abnormally positive investor mood; the market tends to behave much the same regardless of the signal (Chart 11). Sometimes a correction will follow the sentiment highs, whilst many other times the market just continues grinding to new highs. What should we make of the difference relative to the bearish case? It could simply be an artefact of a strong bull market, but I suspect that a deeper asymmetry in stock market behaviour is at play here.

Chart 11. The average cumulative return for the S&P 500 index conditional on abnormally high prior social media sentiment (green line) vs. the unconditional return (blue line). The cumulative return after each of the occasions when the sentiment is high are also shown (grey lines). Sources: Quandl, PsychSignal, own calculations.

At this point, a whole lot of caveats are in order. The findings apply to one provider of sentiment analytics, one country, one specific stock market index and one particular and perhaps atypical time period. This is not a trading strategy: There is no risk control, there are no take-profit levels, and there is no consideration of either the intraday timing of trades or the trading costs. And the analysis is based on a stock index instead of tradable and in any case more efficient stock index futures.

Granting all those caveats, I think that there are a couple of interesting lessons buried in the data.

  • There is a clear asymmetry in bullish vs. bearish investor sentiment. This is evident in the distribution of the stock-level sentiment, in the autocorrelation structure of the aggregate sentiment, and in the degree that the sentiment is influenced by the past price action.
  • Pockets of predictability can be found when the sentiment approaches the extremes. But there’s a difference in how the stock market performs after sentiment peaks and troughs.

In general, the findings suggest that there is something tangible in a concept which is prima facie of such an ethereal and emotional nature. It is quite another matter of how to best take advantage of our ability to track the investor sentiment, and in which instruments and asset classes can the best opportunities be found.


  1. Given the sample size, the 95% confidence interval is about 0.054.
  2. You’d obviously see a different chart if you varied the signal methodology or the threshold. In the case of bearish sentiment, at least, the pattern is fairly robust with respect to parameter variations.

Comments are closed.


© The Behavioural Quant. All rights reserved. Powered by WordPress.