Solicited public opinion surveys reach a limited subpopulation of willing participants and are expensive to conduct, leading to poor time resolution and a restricted pool of expert-chosen survey topics. In this study, we demonstrate that unsolicited public opinion polling through sentiment analysis applied to Twitter correlates well with a range of traditional measures, and has predictive power for issues of global importance. We also examine Twitter's potential to canvas topics seldom surveyed, including ideas, personal feelings, and perceptions of commercial enterprises. Two of our major observations are that appropriately filtered Twitter sentiment (1) predicts President Obama's job approval three months in advance, and (2) correlates well with surveyed consumer sentiment. To make possible a full examination of our work and to enable others' research, we make public over 10,000 data sets, each a seven-year series of daily word counts for tweets containing a frequently used search term.

Since the shooting of Black teenager Michael Brown by White police officer Darren Wilson in Ferguson, Missouri, the protest hashtag #BlackLivesMatter has amplified critiques of extrajudicial killings of Black Americans. In response to #BlackLivesMatter, other Twitter users have adopted #AllLivesMatter, a counter-protest hashtag whose content argues that equal attention should be given to all lives regardless of race. Through a multi-level analysis, we study how these protests and counter-protests diverge by quantifying aspects of their discourse. In particular, we introduce methodology that not only quantifies these divergences, but also reveals whether they are from widespread discussion or a few popular retweets within these groups. We find that #BlackLivesMatter exhibits many informationally rich conversations, while those within #AllLivesMatter are more muted and susceptible to hijacking. We also show that the discussion within #BlackLivesMatter is more likely to center around the deaths of Black Americans, while that of #AllLivesMatter is more likely to sympathize with the lives of police officers and express politically conservative views.

The task of text segmentation, or 'chunking,' may occur at many levels in text analysis, depending on whether it is most beneficial to break it down by paragraphs of a book, sentences of a paragraph, etc. Here, we focus on a fine-grained segmentation task, which we refer to as text partitioning, where we apply methodologies to segment sentences or clauses into phrases, or lexical constructions of one or more words. In the past, we have explored (uniform) stochastic text partitioning—a process on the gaps between words whereby each space assumes one from a binary state of fixed (word binding) or broken (word separating) by some probability. In that work, we narrowly explored perhaps the most naive version of this process: random, or, uniform stochastic partitioning, where all word-word gaps are prescribed a uniformly-set breakage probability, q. Under this framework, the breakage probability is a tunable parameter, and was set to be pure-uniform: q = 1/2. In this work, we explore phrase frequency distributions under variation of the parameter q, and define non-uniform, or informed stochastic partitions, where q is a function of surrounding information. Using a crude but effective function for q, we go on to apply informed partitions to over 20,000 English texts from the Project Gutenberg eBooks database. In these analyses, we connect selection models to generate a notion of fit goodness for the 'bag-of-terms' (words or phrases) representations of texts, and find informed (phrase) partitions to be an improvement over the q = 1 (word) and q = 1/2 (phrase) partitions in most cases. This, together with the scalability of the methods proposed, suggests that the bag-of-phrases model should more often than not be implemented in place of the bag-of-words model, setting the stage for a paradigm shift in feature selection, which lies at the foundation of text analysis methodology.

We propose and develop a Lexicocalorimeter: an online, interactive instrument for measuring the "caloric content" of social media and other large-scale texts. We do so by constructing extensive yet improvable tables of food and activity related phrases, and respectively assigning them with sourced estimates of caloric intake and expenditure. We show that for Twitter, our naive measures of "caloric input", "caloric output", and the ratio of these measures&mdash"caloric balance"&mdashare all strong correlates with health and well-being demographics for the contiguous United States. Our caloric balance measure outperforms both its constituent quantities; is tunable to specific demographic measures such as diabetes rates; provides a real-time signal reflecting a population's health; and has the potential to be used alongside traditional survey data in the development of public policy and collective self-awareness. Because our Lexicocalorimeter is a linear superposition of principled phrase scores, we also show we can move beyond correlations to explore what people talk about in collective detail, and assist in the understanding and explanation of how population-scale conditions vary, a capacity unavailable to black-box type methods.

The Google Books corpus contains millions of books in a variety of languages. Due to its incredible volume and its free availability, it is a treasure trove for linguistic research. In a previous work, we found the unfiltered English data sets from both the 2009 and 2012 versions of the corpus are both heavily saturated with scientific literature, as is the 2009 version of the English Fiction data set. Fortunately, the 2012 version of English Fiction is consistent with fiction and shows promise as an indicator of the evolution of the English language as used by the general public. In this paper, we first critique a method used by authors of an earlier work to determine the birth and death rates of words in a given linguistic data set. We show that this earlier method produces an artificial surge in the death rate at the end of the observed period of time. In order to avoid this boundary effect in our own analysis of asymmetries in language dynamics, we examine the volume of word flux across various relative frequency thresholds for the 2012 English Fiction data set. We then use the contributions of the words crossing these thresholds to the Jensen-Shannon divergence between consecutive decades to resolve the major driving factors behind the flux.

Complex, dynamic networks underlie many systems, and understanding these networks is the concern of a great span of important scientific and engineering problems. Quantitative description is crucial for this understanding yet, due to a range of measurement problems, many real network datasets are incomplete. Here we explore how accidentally missing or deliberately hidden nodes may be detected in networks by the effect of their absence on predictions of the speed with which information flows through the network. We use Symbolic Regression (SR) to learn models relating information flow to network topology. These models show localized, systematic, and non-random discrepancies when applied to test networks with intentionally masked nodes, demonstrating the ability to detect the presence of missing nodes and where in the network those nodes are likely to reside.

Background Twitter has become the “wild-west” of marketing and promotional strategies for advertisement agencies. Electronic cigarettes have been heavily marketed across Twitter feeds, offering discounts, “kid-friendly” flavors, algorithmically generated false testimonials, and free samples.
Methods All electronic cigarette keyword related tweets from a 10% sample of Twitter spanning January 2012 through December 2014 (approximately 850,000 total tweets) were identified and categorized as Automated or Organic by combining a keyword classification and a machine trained Human Detection algorithm. A sentiment analysis using Hedonometrics was performed on Organic tweets to quantify the change in consumer sentiments over time. Commercialized tweets were topically categorized with key phrasal pattern matching.
Results The overwhelming majority (80%) of tweets were classified as automated or promotional in nature. The majority of these tweets were coded as commercialized (83.65% in 2013), up to 33% of which offered discounts or free samples and appeared on over a billion twitter feeds as impressions. The positivity of Organic (human) classified tweets has decreased over time (5.84 in 2013 to 5.77 in 2014) due to a relative increase in the negative words `ban', `tobacco', `doesn't', `drug', `against', `poison', `tax' and a relative decrease in the positive words like `haha', `good', `cool'. Automated tweets are more positive than organic (6.17 versus 5.84) due to a relative increase in the marketing words like `best', `win', `buy', `sale', `health', `discount' and a relative decrease in negative words like `bad', `hate', `stupid', `don't'.
Conclusions Due to the youth presence on Twitter and the clinical uncertainty of the long term health complications of electronic cigarette consumption, the protection of public health warrants scrutiny and potential regulation of social media marketing.

Instabilities and long term shifts in seasons, whether induced by natural drivers or human activities, pose great disruptive threats to ecological, agricultural, and people. Here, we propose, measure, and explore two fundamental markers of location-sensitive seasonal variations: the Summer and Winter Teletherms—the on-average annual dates of the hottest and coldest days of the year. We analyse daily temperature extremes recorded at 1218 stations across the contiguous United States from 1853–2012, and observe large regional variation with the Summer Teletherm falling up to 90 days after the Summer Solstice, and 50 days for the Winter Teletherm after the Winter Solstice. We show that Teletherm temporal dynamics are substantive with clear and in some cases dramatic shifts reflective of system bifurcations. We also compare recorded daily temperature extremes with output from two regional climate models finding considerable though relatively unbiased error. Our work demonstrates that Teletherms are an intuitive, powerful, and statistically sound measure of local climate change, and that they pose detailed, stringent challenges for future theoretical and computational models.

Twitter, a popular social media outlet, has evolved into a vast source of linguistic data, rich with opinion, sentiment, and discussion. Due to the increasing popularity of Twitter, its perceived potential for exerting social influence has led to the rise of a diverse community of automatons, commonly referred to as bots. These inorganic and semi-organic Twitter entities can range from the benevolent (e.g., weather-update bots, help-wanted-alert bots) to the malevolent (e.g., spamming messages, advertisements, or radical opinions). Existing detection algorithms typically leverage meta-data (time between tweets, number of followers, etc.) to identify robotic accounts. Here, we present a powerful classification scheme that exclusively uses the natural language text from organic users to provide a criterion for identifying accounts posting automated messages. Since the classifier operates on text alone, it is flexible and may be applied to any textual data beyond the Twitter-sphere.

In an effort to better understand meaning from natural language texts, we explore methods aimed at organizing lexical objects into contexts. A number of these methods for organization fall into a family defined by word ordering. Unlike demographic or spatial partitions of data, these collocation models are of special importance for their universal applicability in the presence of ordered symbolic data (e.g., text, speech, genes, etc...). Our approach focuses on the phrase (whether word or larger) as the primary meaning-bearing lexical unit and object of study. To do so, we employ our previously developed framework for generating word-conserving phrase-frequency data. Upon training our model with the Wiktionary—an extensive, online, collaborative, and open-source dictionary that contains over 100, 000 phrasal-definitions—we develop highly effective filters for the identification of meaningful, missing phrase-entries. With our predictions we then engage the editorial community of the Wiktionary and propose short lists of potential missing entries for definition, developing a breakthrough, lexical extraction technique, and expanding our knowledge of the defined English lexicon of phrases.

The consequences of anthropogenic climate change are extensively debated through scientific papers, newspaper articles, and blogs. Newspaper articles may lack accuracy, while the severity of findings in scientific papers may be too opaque for the public to understand. Social media, however, is a forum where individuals of diverse backgrounds can share their thoughts and opinions. As consumption shifts from old media to new, Twitter has become a valuable resource for analyzing current events and headline news. In this research, we analyze tweets containing the word “climate” collected between September 2008 and July 2014. We determine how collective sentiment varies in response to climate change news, events, and natural disasters. Words uncovered by our analysis suggest that responses to climate change news are predominately from climate change activists rather than climate change deniers, indicating that Twitter is a valuable resource for the spread of climate change awareness.

The concerns expressed by Garcia et al. are misplaced due to a range of misconceptions about word usage frequency, word rank, and expert-constructed word lists such as LIWC (Linguist Inquiry and Word Count). We provide a complete response in our paper's online appendices.

Using human evaluation of 100,000 words spread across 24 corpora in 10 languages diverse in origin and culture, we present evidence of a deep imprint of human sociality in language, observing that (1) the words of natural human language possess a universal positivity bias; (2) the estimated emotional content of words is consistent between languages under translation; and (3) this positivity bias is strongly independent of frequency of word usage. Alongside these general regularities, we describe inter-language variations in the emotional spectrum of languages which allow us to rank corpora. We also show how our word evaluations can be used to construct physical-like instruments for both real-time and offline measurement of the emotional content of large-scale texts.

Reflective of income and wealth distributions, philanthropic gifting appears to follow an approximate power-law size distribution as measured by the size of gifts received by individual institutions. We explore the ecology of gifting by analysing data sets of individual gifts for a diverse group of institutions dedicated to education, medicine, art, public support, and religion. We find that the detailed forms of gift-size distributions differ across but are relatively constant within charity categories. We construct a model for how a donor's income affects their giving preferences in different charity categories, offering a mechanistic explanation for variations in institutional gift-size distributions. We discuss how knowledge of gift-sized distributions may be used to assess an institution's gift-giving profile, to help set fundraising goals, and to design an institution-specific giving pyramid.

The patterns of life exhibited by large populations have been described and modeled both as a basic science exercise and for a range of applied goals such as reducing automotive congestion, improving disaster response, and even predicting the location of individuals. However, these studies previously had limited access to conversation content, rendering changes in expression as a function of movement invisible. In addition, they typically use the communication between a mobile phone and its nearest antenna tower to infer position, limiting the spatial resolution of the data to the geographical region serviced by each cellphone tower. We use a collection of 37 million geolocated tweets to characterize the movement patterns of 180,000 individuals, taking advantage of several orders of magnitude of increased spatial accuracy relative to previous work. Employing the recently developed sentiment analysis instrument known as the hedonometer, we characterize changes in word usage as a function of movement, and find that expressed happiness increases logarithmically with distance from an individual's average location.

We conduct a detailed investigation of correlations between real-time expressions of individuals made across the United States and a wide range of emotional, geographic, demographic, and health characteristics. We do so by combining (1) a massive, geo-tagged data set comprising over 80 million words generated over the course of several recent years on the social network service Twitter and (2) annually-surveyed characteristics of all 50 states and close to 400 urban populations. Among many results, we generate taxonomies of states and cities based on their similarities in word use; estimate the happiness levels of states and cities; correlate highly-resolved demographic characteristics with happiness levels; and connect word choice and message length with urban characteristics such as education levels and obesity rates. Our results show how social media may potentially be used to estimate real-time levels and changes in population-level measures such as obesity rates.

The advent of social media has provided an extraordinary, if imperfect, 'big data' window into the form and evolution of social networks. Based on nearly 40 million message pairs posted to Twitter between September 2008 and February 2009, we construct and examine the revealed social network structure and dynamics over the time scales of days, weeks, and months. At the level of user behavior, we employ our recently developed hedonometric analysis methods to investigate patterns of sentiment expression. We find users' average happiness scores to be positively and significantly correlated with those of users one, two, and three links away. We strengthen our analysis by proposing and using a null model to test the effect of network topology on the assortativity of happiness. We also find evidence that more well connected users write happier status updates, with a transition occurring around Dunbar's number. More generally, our work provides evidence of a social sub-network structure within Twitter and raises several methodological points of interest with regard to social network reconstructions.

Individual happiness is a fundamental societal metric. Normally measured through self-report, happiness has often been indirectly characterized and overshadowed by more readily quantifiable economic indicators, such as gross domestic product. Here, we use a real-time, remote-sensing, non-invasive, text-based approach&emdash;a kind of hedonometer&emdash;to uncover collective dynamical patterns of happiness levels expressed by over 50 million users in the online, global social network Twitter. With a data set comprising nearly 2.8 billion expressions involving more than 28 billion words, we explore temporal variations in happiness, as well as information levels, over time scales of hours, days, and months. Among many observations, we find a steady global happiness level, evidence of universal weekly and daily patterns of happiness and information, and that happiness and information levels are generally uncorrelated. We also extract and analyse a collection of happiness and information trends based on keywords, showing them to be both sensible and informative, and in effect generating opinion polls without asking questions. Finally, we develop and employ a graphical method that reveals how individual words contribute to changes in average happiness between any two texts.

For a broad range single-seed contagion processes acting on generalized random networks, we derive a unifying analytic expression for the possibility of global spreading events in a straightforward, physically intuitive fashion. Our reasoning lays bare a direct mechanical understanding of an archetypal spreading phenomena that is not evident in circuitous extant mathematical approaches.

We investigate by numerical simulation a threshold model of social contagion on degree-correlated random networks. We show that the class of networks for which global information cascades occur generally expands as degree-degree correlations become increasingly positive. However, under certain conditions, large-scale information cascades can paradoxically occur when degree-degree correlations are sufficiently positive or negative, but not when correlations are relatively small. We also show that the relationship between the degree of the initially infected vertex and its ability to trigger large cascades is strongly affected by degree-degree correlations.