Ethan Chiu My personal blog

Creating Realtime, Client-side Twitter Analysis Script for the United Timeline

I recently started working on a project that shows and analyzes real-time Conservative, Moderate, and Liberal viewpoints across social media called the United Timeline.

Here’s the mission statement: “The media constantly bombards us with polarizing rhetoric that support our own viewpoints. Without viewing other perspectives, we create an echo chamber that limits the extent to which we can have an open dialogue that critically engages with important topics and situations. Ignoring arguments from the other political side creates divisiveness, furthering the divide between Americans and bringing no real progress when it comes to reforms and policies in Congress and elsewhere.

The United Timeline provides real-time social media posts and analysis of liberal, moderate, and conservative viewpoints. Our goal is to bridge the divide between liberals, conservatives, and moderates by sharing the current conversation from each side on social media.”

Currently, we show what Twitter news feeds would look like from those different viewpoints.To achieve this, I started out creating the Twitter timelines for Conservative, Moderate, and Liberal viewpoints. This was pretty straightforward. I created a Twitter account and created three different lists containing the most prominent and intelligent pundits from each side. Then, I simply embedded them side by side.

After that, I wanted to generate word clouds of the most recent Tweets. One way that instantly popped up in my mind was to simply use Twitter’s API. Unfortunately, the API has limits that would easily be passed for real-time analysis. Another method was using Selenium to constantly scrape the most recent Tweets. This would require me to create a server with a backend and also might not work since Twitter blocks scraping after a certain limit.

So, I created a way to analyze the 20 most recent Tweets from each list all in the client-side (no backend). As a result, I don’t have to rely on backend processing or worry about server costs. To analyze these Tweets in real time, I first analyzed how Twitter’s embedded lists load onto a site. I saw that the embedded link transformed into a asynchronous script, meaning it loads in the background of the website as the user interacts with the list. As a result, I programmed a script which checks when the async Twitter script is loaded for each list. Afterwards, it grabs the top 20 Tweets’ text content using jQuery to identify the class names. Then, I use regex to eliminate any urls, @ tags, or punctuation so that only words are generated in the word clouds. After that, I calculate the word frequency and format that into a list that the wordcloud2 Javascript library can understand.

So, my script allows the user to see realtime graphs of the 20 most recent Tweets from each list every time he or she refreshes the page!

Here’s the code:

$( document ).ready(function() {
	var $canvas = $('#word_cloud');
	generateWordCloud();
});

//Edited https://stackoverflow.com/questions/30906807/word-frequency-in-javascript
function wordFreq(string) {
    var words = string.replace(/[.]/g, '').split(/\s/);
    var freqMap = {};
    words.forEach(function(w) {
        if (!freqMap[w]) {
            freqMap[w] = 0;
        }
        freqMap[w] += 1;
    });

    return freqMap;
}

function generateWordCloud(){
	setTimeout(
	    function() {
	    	//Set up variables, get document values from iframes loaded asyncronously
	    	var count = [];
	    	var liberal = document.getElementById('twitter-widget-0').contentWindow.document;
	    	var moderate = document.getElementById('twitter-widget-1').contentWindow.document;
	    	var conservative = document.getElementById('twitter-widget-2').contentWindow.document;
	    	// console.log(liberal)

	    	//Get Tweets from each iframe
	    	liberalTweetsHTML = $(liberal).find(".timeline-Tweet-text");
	    	count.push(liberalTweetsHTML.length);
	    	moderateTweetsHTML = $(moderate).find(".timeline-Tweet-text");
	    	count.push(moderateTweetsHTML.length);
	    	conservativeTweetsHTML = $(conservative).find(".timeline-Tweet-text");
	    	count.push(conservativeTweetsHTML.length);

	    	// console.log(count);
	    	//If there are less than 10 tweets for any of the feeds, recursively run this function until there are 10 tweets.
	    	if(count[0] < 10 || count[1] < 10 || count[2] < 10)
	    	{
	    		generateWordCloud();
	    		return;
	    	}
	    	
	    	//Extract text from Tweets
	    	var liberalTweets =[];
	    	for(var i = 0; i<liberalTweetsHTML.length; i++){
	    		liberalTweets.push(liberalTweetsHTML[i].innerText);
	    	}
	    	var moderateTweets =[];
	    	for(var i = 0; i<moderateTweetsHTML.length; i++){
	    		moderateTweets.push(moderateTweetsHTML[i].innerText);
	    	}
	    	var conservativeTweets =[];
	    	for(var i = 0; i<conservativeTweetsHTML.length; i++){
	    		conservativeTweets.push(conservativeTweetsHTML[i].innerText);
	    	}
	    	// console.log(liberalTweets);
	    	// console.log(moderateTweets);
	    	// console.log(conservativeTweets);

	    	//Generate Word Clouds
	    	//Regex to elimate tags, etc. + join words into one coherent group
	    	liberalTweets = liberalTweets.join(' ');
	    	moderateTweets = moderateTweets.join(' ');
	    	conservativeTweets = conservativeTweets.join(' ');

	    	// console.log(liberalTweets);
	    	//Get rid of @ mentions
	    	liberalTweets = liberalTweets.replace(/\S*@\S*\s?/g, "");
	    	//Get rid of links
	    	liberalTweets = liberalTweets.replace(/(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?/g,"");
	    	// console.log(liberalTweets);
	    	//Get rid of punctionation + spaces
	    	var punctuationless = liberalTweets.replace(/[.,\/#!$%\^&\*;:{}=\-_`~()]/g,"");
			liberalTweets = punctuationless.replace(/\s{2,}/g," ");
			//Get rid of articles of speech
			regex = /(?:(the|a|an|and|of|what|to|for|about) +)/g; 
			liberalTweets = liberalTweets.replace(regex, "");
	    	// liberalTweets = liberalTweets.split(" ");
	    	// console.log(liberalTweets);
	    	listA = wordFreq(liberalTweets);
	    	var list = [];
	    	for (var key in listA)
	    	{
	    		list.push([key, listA[key]*5]);
	    	}
	    	console.log(list);
			console.log(WordCloud.isSupported);
			WordCloud.minFontSize = "15px";
			WordCloud($('#liberal_word_cloud')[0], { list: list });

			moderateTweets = moderateTweets.replace(/\S*@\S*\s?/g, "");
	    	//Get rid of links
	    	moderateTweets = moderateTweets.replace(/(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?/g,"");
	    	// console.log(liberalTweets);
	    	//Get rid of punctionation + spaces
	    	var punctuationless = moderateTweets.replace(/[.,\/#!$%\^&\*;:{}=\-_`~()]/g,"");
			moderateTweets = punctuationless.replace(/\s{2,}/g," ");
			//Get rid of articles of speech
			regex = /(?:(the|a|an|of|and|what|to|for|about) +)/g; 
			moderateTweets = moderateTweets.replace(regex, "");
	    	// liberalTweets = liberalTweets.split(" ");
	    	// console.log(liberalTweets);
	    	listA = wordFreq(moderateTweets);
	    	var list = [];
	    	for (var key in listA)
	    	{
	    		list.push([key, listA[key]*5]);
	    	}
	    	console.log(list);
			console.log(WordCloud.isSupported);
			WordCloud.minFontSize = "15px";
			WordCloud($('#moderate_word_cloud')[0], { list: list });

			conservativeTweets = conservativeTweets.replace(/\S*@\S*\s?/g, "");
	    	//Get rid of links
	    	conservativeTweets = conservativeTweets.replace(/(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?/g,"");
	    	// console.log(liberalTweets);
	    	//Get rid of punctionation + spaces
	    	var punctuationless = conservativeTweets.replace(/[.,\/#!$%\^&\*;:{}=\-_`~()]/g,"");
			conservativeTweets = punctuationless.replace(/\s{2,}/g," ");
			//Get rid of articles of speech
			regex = /(?:(the|a|an|and|of|what|to|for|about) +)/g; 
			conservativeTweets = conservativeTweets.replace(regex, "");
	    	// liberalTweets = liberalTweets.split(" ");
	    	// console.log(liberalTweets);
	    	listA = wordFreq(conservativeTweets);
	    	var list = [];
	    	for (var key in listA)
	    	{
	    		list.push([key, listA[key]*5]);
	    	}
	    	console.log(list);
			console.log(WordCloud.isSupported);
			WordCloud.minFontSize = "15px";
			WordCloud($('#conservative_word_cloud')[0], { list: list });
	    }, 250);
}

Here’s the link where you can see this all in action. Screenshot United Timeline Analysis

Obviously, this isn’t the prettiest solution out there. I could of used recursion and for loops to eliminate some repetition, but I feel like this is the clearest explanation. For a full list of my other attempts for analyzing the Tweets in realtime completely client-side, please check out the Javascript file on Github.

Since this script only works for 20 of the most recent Tweets, I’m most likely going to create a Python backend to process more Tweets (like every Tweet in a day) and present different word clouds daily. This would also allow me to use libraries such as NTLK to filter out connecting words.

We are currently working on adding more analytic tools and integrations. This website is completely open source and open to contributions. We are currently working on adding more analytic tools and integrations. If you have any suggestions/concerns, feel free to open up an issue over here.