Reading the past like an open book: Researchers use text to measure 200 years of happiness
Using innovative new methods researchers at the University of Warwick, University of Glasgow Adam Smith Business School and The Alan Turing Institute in London have built a new index that uses data from books and newspaper to track levels of national happiness from 1820. Their research could help governments to make better decisions about policy priorities.
Governments the world over are making increasing use of “national happiness” data derived from surveys to help them consider the impact of policy on national wellbeing. Unfortunately, data for most countries is only available from 2011 onwards, and for a select few from the mid 1970s. This makes it hard to establish long-run trends, or to say anything about the main historical causes of happiness.
In order to tackle this problem, a team of researchers including Professor Thomas Hills (Warwick and The Alan Turing Institute), Professor Eugenio Proto (Glasgow), Professor Daniel Sgroi (Warwick), and Dr Chanuki Seresinhe (The Alan Turing Institute) took a key insight from psychology — that more often than not what people say or write reveals much about their underlying happiness level — and developed a method to apply it to online texts from millions of books and newspapers published over the past 200 years.
The main source of language used for the analysis was the Google Books corpus, a collection of word frequency data for over 8 million books — that’s more than 6 per cent of all books ever published.
The method uses psychological valence norms — values of happiness that can be derived from text — for thousands of words in di?erent languages to compute the relative proportion of positive and negative language for four di?erent nations (the USA, UK, Germany and Italy). The research team also controlled for the evolution of language, to take into account the fact that some words change their meaning over time.
The new index was validated against existing survey-based measures and proven to be an accurate guide to the national mood. One theory as to why books and newspaper articles are such a good source of data is that editors prefer to publish pieces which match the mood of their readers.
Studying the index, the researchers found that:
Commenting on the findings, Professor Thomas Hills said: “What’s remarkable is that national subjective well-being is incredibly resilient to wars. Even temporary economic booms and busts have little long-term effect. We can see the American Civil War in our data, the revolutions of 48′ across Europe, the roaring 20’s and the Great Depression. But people quickly returned to their previous levels of subjective well-being after these events were over. Our national happiness is like an adjustable spanner that we open and close to calibrate our experiences against our recent past, with little lasting memory for the triumphs and tragedies of our age.”
Professor Eugenio Proto added: “Our index is an important first step in understanding people’s satisfaction in the past. Looking at the Italian data, it is interesting to note a slow but constant decline in the years of fascism and a dramatic decline in the years after the last crisis.”
Professor Daniel Sgroi said: ‘Aspirations seem to matter a lot: after the end of rationing in the 1950s national happiness was very high as were expectations for the future, but unfortunately things did not pan out as people might have hoped and national happiness fell for many years until the low-point of the Winter of Discontent.’
Dr Chanuki Seresinhe said: “It was really important to ensure that the changing meaning of words over time was taken into account. For example, the word “gay” had a completely different meaning in the 1800s than it does today. We processed terabytes of word co-occurrence data from Google Books to understand how the meaning of words has changed over time, and we validate our findings using only words with the most stable historical meanings.”