Random Quote

Wit is the unexpected copulation of ideas. — Patrick O’Brian


Enter your email address below to receive notifications of new posts.

Investor sentiment and the stock market

Social media and its relevance to quantitative investing is something that has occupied much of my time recently. I realise that it is a dangerous thing for one’s professional reputation to write about something where hype is the norm rather than the exception. But let me put it this way: Imagine that you are a driver on a long and perilous journey along a narrow and winding road in a mountain country. The road is under a variable but heavy fog cover, so that your headlights are of not much use in seeing the next turn ahead. You come to a service station, and on the shelf you see a device that uses infrared radiation to produce a live image of the road ahead. Being of a sceptical mindset, you suspect that — despite shiny packaging and far-reaching claims on the cover — the device may not be any good in practise. But — just perhaps — it is worth taking it along for a test drive. After all, this is a road where you really want any additional source of illumination you can get.

It is in this light that I see the role of social media analytics in investing. Social media provide one more data source that can shed light on investor sentiment and can help identify the focus of market attention. It may well be the case that much of the data is pure noise and has no value on most days and for most assets, or it may capture only information that can be obtained from other existing sources. But then again, why not have a look at the new data to see if it adds anything? The potential rewards from improved prediction can be high, and — as long as you don’t rely on social media alone — not much is lost beyond the time spent on research and some expenses.

I’ll explore social media analytics in a bit more detail in my next post. If you bear with me, I’d first like to place social media into a broader context among a variety of investor sentiment indicators.

Investor sentiment is not a necessarily irrational concept. When times are good, investors have a sanguine and justifiably positive view about the economy and the stock market. And in bad times, investors are gloomy with good reason. A more relevant question is whether the sentiment in a point in time is already fully reflected in asset prices, or whether the sentiment has taken the price too far in one direction and away from the fundamental value. There certainly are many examples in history where asset prices have either reached irrational levels or changed quickly and dramatically without an adequate rational explanation. We only need to think back a few years to the global financial crisis in 2008, or the dot-com bubble and the subsequent crash in 2000, or earlier episodes such the Black Monday in October 1987, the Nifty Fifty in the early 1970’s, or the Great Crash in 1929, to pick just a few examples. The classic treatment of manias, panics and crashes is still the work by Charles Kindleberger and Robert Aliber1.

The significance of investor sentiment has been long recognised by practitioners and academics alike. Many different measures of investor sentiment have been used either as part of the investment process or as raw data for academic study. What is common to all of the proposed measures is the fact that they are all proxies. Unfortunate as this is, there’s no method yet of directly measuring the sentiment or emotions of large and geographically diverse groups of people. Anecdotally, classical economists like Francis Ysidro Edgeworth, Frank Ramsey and Irving Fisher clearly saw the need for a device — a “hedonimeter” or a “psychogalvanometer” — that could measure the all-important concept of economic utility directly. Colin Camerer quotes Edgeworth2 on this:

…imagine an ideally perfect instrument, a psychophysical machine, continually registering the height of pleasure experienced by an individual… From moment to moment the hedonimeter varies; the delicate index now flickering with the flutter of the passions, now steadied by intellectual activity, low sunk whole hours in the  neighborhood of zero, or momentarily springing up towards infinity…

The emerging field of neuroeconomics may one day — at least in a laboratory setting — give us the means of measuring sentiment directly. For now, we must settle for a proxy in want of the true thing. Here’s a list of possible proxy measures3:

Surveys. You can simply ask investors about their views on security prices. Investors Intelligence conducts weekly surveys of newsletter writers and economic advisers, and has been in the business since 1960’s. American Association of Individual Investors (AAII) conducts weekly surveys of retail investors and publishes bullish/neutral/bearish readings. Federal Reserve targets market professionals in its senior local officers’ survey, which looks at the proportion of banks tightening credit to large and medium firms.

Mood variables. Researchers have looked at specific mood variables such as the time of the year4, football results5 and the morning sunshine6, and have found patterns suggestive of a correlation between the sentiment and the stock market returns.

Mutual fund flows. There is plentiful data available of investor flows in and out of fund in different sectors, e.g. stock funds, bond funds, or the money market funds. Sources include companies such as Lipper and Morningstar. Flow data is typically available at a weekly frequency. To the extent that investors’ allocation decisions reflect their sentiment, fund flows convey useful information.

Trading volume. Although trading volume is readily available in real-time or with only minimal delay in most markets, the interpretation is not straight-forward. Heavy volume can be a sign of either positive or negative views on a security, or a sign of significant differences in opinion. It appears to me that trading volume can only be useful as a sentiment proxy if we make the assumption that costly short-selling biases volume towards times when investors are optimistic.

Dividend premium. This is the difference between the average book-to-market ratio for dividend-paying vs. non-dividend-paying stocks. The motivation is that investors are likely to pay more for “safe” dividend-paying stocks when they are worried about the market.

Closed-end fund discount. There are investment companies which are listed and trade publicly, having issued a fixed number of shares. If such funds are held largely by retail investors, the discount (or premium) of the fund price relative to its net asset value can be viewed as a sentiment proxy relevant to the fund’s investment universe.

IPO activity. If stocks in initial public offerings earn a sizeable first-day return, then either exuberant investor sentiment or severe underpricing is at play. And IPO volume exhibits significant variability, presumably because investment bankers make an effort to assess the public’s appetite for new issues.

Insider trading. Corporate executives typically have more insight than the investing public as to the fairness of the company’s stock price. If market sentiment lifts (or sinks) all boats, mispricing will follow, and readily available data about executives’ personal transactions in the stock conveys sentiment information.

The above measures of sentiment represent the state of the art before the widespread use of the Internet and the era of the social media. Things are different now with the digital revolution bringing in whole news ways of measuring investor sentiment. Below, I’ll mention some of the avenues that are being explored.  I will write more about these topics on another day.

Media and the stock market. There’s now a good and still growing body of research on media and the stock market. Instead of simply assuming that markets efficiently incorporate news into asset prices, a number of researchers have decided to get their hands dirty and actually have a look at the impact of news. This research has produced some interesting results. For instance, it has been shown that country-specific news on the front page of New York Times affect the closed-end country fund discount7. It has also been found that there’s a difference in price momentum and reversal patterns following large moves with and without relevant news 8, media pessimism predicts downward pressure in stock prices 9, and individual investors are net buyers of stocks which are prominent in the news10. However, it also appears that stocks with no media coverage earn higher returns than those with high coverage11. Importantly, there is a causal link between media reporting and investor actions12, given that it is the local media coverage which predicts local trading in the stock market.

Search activity. Internet is a wonderful thing, but it would not be particularly useful without efficient means of navigating it and finding the information you need. And this is, of course, where search engines come in. There are many such engines, with some specialised in a particular field or specific kind of information. Others — like Yahoo, Bing, and Google — are good for general enquiries. Of these, Google is by far the most popular. For our purposes, the aggregate search activity is interesting because it is a direct and revealed measure of public attention. Web search data has already proved useful in many fields. For instance, Google search volume can “predict the present” for statistics such as unemployment13 and home sales14. It has also been demonstrated that search volume can predict the future, improving forecasts for consumer behaviour days and even weeks ahead15.

In finance, it has been found that Google search volume is correlated with other sentiment measures but it is more timely. An increase in investor attention predicts higher stock prices and eventual reversals16. As an other example, it looks like a heightened number of stock market searches today results in increased volatility tomorrow17. There is also a somewhat controversial study which suggests that an increase in Google search volume for risk-related keywords precedes market falls, and a decrease in search volume for the same precedes price rises18. It does appear, though, as if a high volume of queries for household concerns is associated with low stock market returns today and high returns tomorrow19.

Social media. Automated social media analytics provide a new and timely way to measure market sentiment. As of this writing, the social media phenomenon is not much more than a decade old. Facebook was established in 2004, and there are now over 1.35 billion active users on it, with some 864 million people logging on daily as of October 2014. And it is not the only game in town. If you count only those with more than 100 million users, there are now 12 significant social media platforms out there20. In finance, the Twitter micro-blogging platform is the most popular, but blogging platforms — such as Tumblr and WordPress — have a wide following too.

Finance has been slower in embracing social media than many other professions21 but the situation has changed markedly in the last two years. There have been notable examples about the impact that the social media can have on market movements. It is also becoming apparent that social media does provide a new window into the thinking of market participants. It can also indicate where the market focus lies at any particular time. The amplification mechanisms in the social media are the key element in this: Tweets are retweeted, and information is thus propagated through the investor community. The day may come when checking the Twitter and other social media data on a company is as commonplace as checking its security price. For now, we are still in a phase where investors, traders and providers of sentiment analytics are innovating and trying to figure out the best ways to use the new data.


  1. Charles P. Kindleberger and Robert Z. Aliber: Manias, Panics and Crashes: A History of Financial Crises, Palgrave Macmillan (2005)
  2. Colin Camerer: Neuroeconomics: Using neuroscience to make economic predictions, Economic Journal, Vol. 117, No. 519, pp. C26-C42 (March 2007)
  3. Malcolm Baker and Jeffrey Wurgler: Investor Sentiment in the Stock Market, The Journal of Economic Perspectives, Vol. 21, No. 2, pp. 129-151 (Spring, 2007)
  4. Mark Kamstra, Lisa Kramer and Maurice Levi: Winter Blues: A SAD Stock Market Cycle, Federal Reserve Bank of Atlanta Working Paper No. 2002-13a (2003)
  5. Alex Edmans, Diego Garcia and Oyvind Norli: Sports Sentiment and Stock Returns, The Journal of Finance, Vol. 62, No. 4, pp. 1967-1998 (2007)
  6. David Hirshleifer and Tyler Shumway: Good Day Sunshine: Stock Returns and the Weather, The Journal of Finance, Vol. 58, pp. 1009–1032 (2003)
  7. Peter Klibanoff, Owen Lamont, and Thierry Wizman, Thierry: Investor Reaction to Salient News in Closed-End Country Funds, The Journal of Finance, Volume 53, No. 2, pp. 673–699 (1998)
  8. Wesley Chan: Stock Price Reaction to News and No-News: Drift and Reversal After Headlines, Journal of Financial Economics, Vol. 70, pp. 223-260 (2003)
  9. Paul Tetlock: Giving Content to Investor Sentiment: The Role of Media in the Stock Market, The Journal of Finance, Vol. 62, pp. 1139-1168 (2007)
  10. Brad Barber and Terrance Odean: All That Glitters: The Effect of Attention and News on the Buying Behavior of Individual and Institutional Investors, Review of Financial Studies, 21(2), pp. 785-818 (2008)
  11. Lily Fang and Joel Peress: Media Coverage and the Cross-Section of Stock Returns, The Journal of Finance, Vol. 64, No. 5, pp. 2023-2052 (2009)
  12. Joseph Engelberg and Christopher Parsons: The Causal Impact of Media in Financial Markets, The Journal of Finance, Vol. 66, No. 1, pp. 67–97 (2011)
  13. Hyunyoung Choi and Hal Varian: Predicting Initial Claims for Unemployment Benefits, Google technical report (2009)
  14. Lynn Wu and Erik Brynjolfsson: The Future of Prediction: How Google Searches Foreshadow Housing Prices and Sales, ICIS 2009 Proceedings, paper 147 (2009)
  15. Sharad Goel et al.: Predicting consumer behavior with Web search, PNAS, vol. 107, No. 41 (October 12, 2010)
  16. Zhi Da, Joseph Engelbert, and Pengjie Gao: In Search of Attention, The Journal of Finance, Vol. 66, No. 5 (2011)
  17. Thomas Dimpfl and Stephan Jank: Can internet search queries help to predict stock market volatility?, Finance Meeting EUROFIDAI-AFFI, Paris (December 2012)
  18. Tobias Preis et al.: Quantifying Trading Behavior in Financial Markets Using Google Trends, Nature, Scientific Reports, Vol. 3, Article No. 1684 (2013)
  19. Zhi Da, Joseph Engelbert, and Pengjie Gao: The Sum of All FEARS: Investor Sentiment and Asset Prices, Review of Financial Studies, Vol 28, 1-32 (2015)
  20. Jeremy Waite: Which Social Networks Should You Care About in 2014? Digital Marketing Blog, Adobe (3 January 2014)
  21. Social Media and the Markets — The Coming of Age, GNIP white paper (2014)

Comments are closed.


© The Behavioural Quant. All rights reserved. Powered by WordPress.