Archive for November, 2009

 

Managing Friends and Followers with Tweetsharp – Part 3 – MultiThreading Tweetsharp for the User Interface

If you haven’t yet please see:
Part 1
Part 2

Next, we want to build some UI to show us the data we’ve got back from Twitter. For this we’re going to use WPF. Before we get started with that, however, a bit more refactoring is needed. Since we want our UI to be nice and responsive, we’re going to do all of our talking to twitter on background threads. To that end, we want to make our TweetSharp code thread safe. We have basically have two major requests that we need to make, one to get all of our friends and one to get all of our followers. We can send those off more or less simultaneously, and wait for twitter to send back the response, all the while keeping the UI thread free to repaint the form as needed.

To that end, let’s wrap our twitter code up into a class. This also has the nice side-effect of limiting our dependency on TweetSharp from the UI layer. We could sprinkle TweetSharp calls throughout the UI, but that would be a refactoring nightmare should Daniel and I ever fall off a cliff and stop updating it, forcing you to move to one of the other (less fun ;) ) libraries out there.

I’m not feeling particularly creative, so I’m going to just call the class “Twitter”:

using System;
using System.Linq;
using System.Collections.Generic;
using Dimebrain.TweetSharp.Extensions;
using Dimebrain.TweetSharp.Fluent;
using Dimebrain.TweetSharp.Model;

namespace Diller.SocialGraphMinder
{
    internal class Twitter
    {
        private readonly string _userName;
        private readonly string _password;
        private TwitterUser _User;

        public Twitter( string username, string password )
        {
           _userName = username;
           _password = password;
           IsAuthenticated = Authenticate(); 
        }
        
        private bool Authenticate()
        {
            var twitter = FluentTwitter.CreateRequest()
                .AuthenticateAs(_userName, _password)
                .Account().VerifyCredentials()
                .AsJson();

            var response = twitter.Request();
           _User = response.AsUser();
           return _User != null; 
        }
        
        public bool IsAuthenticated{ get; private set; }
   }
}

Nothing particularly interesting there. Upon construction, the class performs a credentials verification with the given credentials and sets the IsAuthenticated bool so we know it’s safe to proceed.

Next, we’ll add the friend and follower fetching code:

//add these private members
private IEnumerable<TwitterUser> _friends;
private IEnumerable<TwitterUser> _followers; 

//add these methods
public IEnumerable<TwitterUser> GetFriends()
{
    if (!IsAuthenticated)
    {
       throw new InvalidOperationException("Not authenticated");
    }
    if ( _friends == null ) //only do this if we don't already have the friends list
    {
        var twitter = FluentTwitter.CreateRequest()
                    .AuthenticateAs(_userName, _password)
                    .Configuration.UseGzipCompression() //now using compression for performance
                    .Users().GetFriends().For(_userName)
                    .CreateCursor()
                    .AsJson();
         _friends = GetAllCursorValues(twitter, s => s.AsUsers());
    }
    return _friends; //return either the newly fetched list, or the cached copy
}

public IEnumerable<TwitterUser> GetFollowers()
{
      if ( !IsAuthenticated)
      {
           throw new InvalidOperationException("Not authenticated");
      }
      if (_followers == null)
      {
           var twitter = FluentTwitter.CreateRequest()
                            .AuthenticateAs(_userName, _password)
                            .Configuration.UseGzipCompression()
                            .Users().GetFollowers().For(_userName)
                            .AsJson();

           _followers = GetAllCursorValues(twitter, s => s.AsUsers());
      }
      return _followers; 
}

//we also need the cursor paging code from part 2
private static IEnumerable<T> GetAllCursorValues<T>(ITwitterLeafNode twitter, Func<string, IEnumerable<T>> conversionMethod)
{
   long? nextCursor = -1 ;
   var ret = new List<T>();
   do
   {
       twitter.Root.Parameters.Cursor = nextCursor; 
       var response = twitter.Request();
       IEnumerable<T> values = conversionMethod(response);
       if (values != null)
       {
           ret.AddRange(values);
       }
       nextCursor = response.AsNextCursor();
   } while (nextCursor.HasValue &amp;amp;amp;&amp;amp;amp; nextCursor.Value != 0);
   return ret; 
}

Finally, lets add some methods to get the data we’re really after, the aforementioned ‘spammers’(users who follow us, but whom we don’t follow), and ‘jerks’ (users whom we follow, but who do not follow us back).

public IEnumerable<TwitterUser> GetFriendsWhoDontFollowBack()
{
       friends = GetFriends();
       followers = GetFollowers();
       return friends.Except(followers); 
}

public IEnumerable<TwitterUser> GetFollowersWhoArentFriends()
{
     friends = GetFriends();
     followers = GetFollowers();
     return followers.Except(friends);
}

If we left this code as-is, it would work perfectly well in a single-threaded application. Whatever code path got to the ‘GetFriends’ and ‘GetFollowers’ methods first would make all the proper calls and cache the results in the member variables. However if you try to use this class in a multi-threaded fashion you’re going to be in for a world of hurt. (Well, really, it most likely just plain ol’ won’t work).

If, for example, you create two threads, one to call the ‘GetFriendsWhoDontFollowBack’ method and the other to call the ‘GetFollowersWhoArentFriends’ method, both are going to call GetFriends() and GetFollowers(), likely before the other one has finished. At best, you’re duplicating work and chewing up your API limit unnecessarily, at worst, you’re going to confuse twitter by resetting the cursor on the second thread somewhere in the middle of paging through it on the first, causing it to throw HTTP 500 errors. (This is what happened when I tried it, and it’s something that probably warrants its own blog post as it was unexpected).

So, to remedy this, we need to synchronize access to the TweetSharp calls so we’re not making a tangly mess:

//add the following private members
private readonly object _friendsLock = new object();
private readonly object _followersLock = new object();

//amend the getfriends and getfollowers methods so they look like this: 
public IEnumerable<TwitterUser> GetFriends()
{
    if (!IsAuthenticated)
    {
        throw new InvalidOperationException("Not authenticated");
    }
    //see if friends are already fetched
    if (_friends == null) 
    {
        //if not already fetched
        //wait for exclusive access to this code
        lock (_friendsLock)
        {
            //need to double-check here
            //another thread might have set the member
            //while we were waiting
            if (_friends == null) 
            {
                var twitter = FluentTwitter.CreateRequest()
                    .AuthenticateAs(_userName, _password)
                    .Configuration.UseGzipCompression()
                    .Users().GetFriends().For(_userName)
                    .CreateCursor()
                    .AsJson();

                _friends = GetAllCursorValues(twitter, s => s.AsUsers());
            }
        }//lock is released here
    }
    return _friends; 
}

//make the same adjustments to this method:
public IEnumerable<TwitterUser> GetFollowers()
{
    if ( !IsAuthenticated)
    {
        throw new InvalidOperationException("Not authenticated");
    }
    if (_followers == null)
    {
        lock (_followersLock)
        {
            if (_followers == null)
            {
                var twitter = FluentTwitter.CreateRequest()
                    .AuthenticateAs(_userName, _password)
                    .Configuration.UseGzipCompression()
                    .Users().GetFollowers().For(_userName)
                    .AsJson();

                _followers = GetAllCursorValues(twitter, s => s.AsUsers());
            }
        }
    }
    return _followers; 
}

A few important points:

  • We used separate lock objects around the fetching of _friends and _followers objects as they are independent and can be fetched simultaneously.
  • For best performance we only lock when we’re going to do the fetch. Once the objects exist they can be read without locking by any thread.
  • The double-check for null after acquiring the lock is important to prevent duplication of effort.
  • If we decide to periodically update the _friends and _followers lists, we would probably want to convert our locking behavior to a ReaderWriterLockSlim type, which is ideal for resources that are read a lot but infrequently updated.

It’s also worth noting that Tweetsharp instances aren’t thread safe. You should never try to use the same instance of a FluentTwitter request across multiple threads without doing your own synchronization. Since they are designed to be transient, it’s best to just create a new one on each thread that you’re using to talk to Twitter and leave it there.

This post has gotten too long already, so let’s end it here and take it up again in part 4…at which point, I promise, we will actually put pixels on the screen.

Posted by Jason under General  •  No Comments

Manage your Twitter Friends and Followers with Tweetsharp – Part 2 – Refactoring

In Part 1 we covered getting all of our friends and followers then using LINQ to find users who appeared in only one or the other collections.

However, the code to fetch followers and friends had a lot of duplication in it because they both use cursors to page through the results, and require a loop and some cursor management code to make sure we get all the results.

We can take care of that duplication by moving all the cursor stuff into a generic method that uses its generic type parameter a lambda expression to take care of the context-specific stuff:

static IEnumerable<T> GetAllCursorValues<T>(ITwitterLeafNode twitter, Func<string, IEnumerable<T>> conversionMethod)
{
    long? nextCursor = -1 ;
    var ret = new List<T>();
    do
    {
        twitter.Root.Parameters.Cursor = nextCursor; 
        var response = twitter.Request();
        IEnumerable<T> values = conversionMethod(response);
        if (values != null)
        {
            ret.AddRange(values);
        }
        nextCursor = response.AsNextCursor();
    } while (nextCursor.HasValue && nextCursor.Value != 0);
    return ret; 
}

This method will work with any of the APIs that use cursors. The type parameter T represents the type of object you expect to get back (as an IEnumerable) -in our case we’re expecting TwitterUser, and the conversionMethod is the method that deserializes the response into the expected format – in our case we we’re using the AsUsers method.

Note that we’ve “opted-out” of the fluent nature of the Tweetsharp interface and we’re setting the Cursor property directly on the Parameters object of the root FluentTwitter instance.

So, the code to get friends and followers is now a lot cleaner, we can omit all of the cursor stuff from it and use our new method to handle that:

//create the tweetsharp request to get followers
//omit any cursor stuff as that's handled by the GetAllCursorValues method
var getFollowersReq = FluentTwitter.CreateRequest()
                .AuthenticateAs(_userName, _password)
                .Users().GetFollowers()
                .AsJson();

//call our new method to get all of the values back - will make multiple calls if necessary
var followers = GetAllCursorValues(getFollowersReq , response => response.AsUsers());

//do the same to get friends. 
var getFriendsReq = FluentTwitter.CreateRequest()
                .AuthenticateAs(_userName, _password)
                .Users().GetFriends()
                .AsJson();
var followers = GetAllCursorValues(getFriendsReq , response => response.AsUsers());

//again, use the LINQ extension methods on IEnumerable<T> to get the differences between the two lists
var jerks = friends.Except(followers); 
var spammers = followers.Except(friends); 

Much nicer.

Now that that’s taken care of, we need some user interface – we’ll do that in Part 3.

Posted by Jason under General  •  No Comments

Manage your Twitter Friends and Followers with Tweetsharp – Part 1

I’ve been looking for a better way to keep track of my Twitter friends and followers for a while now. I’m particularly bad at not following back interesting people who follow me because the emails from Twitter don’t contain the user’s bio information and I’m often too busy or distracted to go and login to the twitter.com to see if the new follower merits a follow-back (i.e. isn’t a fembot or some kind of “social media expert” promising to bring me riches, or a billion followers, or a billion rich followers).

To that end, I’m working on a small utility app explicitly for managing your social graph, which I will blog about as I build it, then ultimately make available as a download.

So, for part 1, I’m going to start with finding users who follow me, but who I don’t follow back. This is pretty easy to do with a few Tweetsharp calls and a bit of LINQ.

First, I’ll get all of my friends (people I follow):

long? nextCursor = -1; //creates a new cursor the first time we call GetCursor()
var friends = new List<TwitterUser>();
do
{
    var twitter = FluentTwitter.CreateRequest()
                    .AuthenticateAs(_userName, _password)
                    .Users().GetFriends()
                    .GetCursor(nextCursor.Value)
                    .AsJson();

    var response = twitter.Request();
    var users = response.AsUsers();
    if (users != null)
    {
        friends.AddRange(users);
    }
    nextCursor = response.AsNextCursor();
} while (nextCursor.HasValue &amp;amp;&amp;amp; nextCursor.Value != 0);

The code to get followers is almost identical, except we call ‘GetFollowers’ instead of ‘GetFriends':

long? nextCursor = -1; //creates a new cursor the first time we call GetCursor()
var followers= new List<TwitterUser>();
do
{
    var twitter = FluentTwitter.CreateRequest()
                    .AuthenticateAs(_userName, _password)
                    .Users().GetFollowers()
                    .GetCursor(nextCursor.Value)
                    .AsJson();

    var response = twitter.Request();
    var users = response.AsUsers();
    if (users != null)
    {
        followers.AddRange(users);
    }
    nextCursor = response.AsNextCursor();
} while (nextCursor.HasValue &amp;amp;&amp;amp; nextCursor.Value != 0);

At this point we have two TwitterUser collections, one containing friends and the other containing our followers. We can use LINQ’s extension methods to find the objects that exist in only one or the other collection with a single call:

//get people we follow who don't follow us back (how rude)
var jerks = friends.Except(followers);  //(I <3 LINQ for methods like this) 

//get people who follow us, but don't follow back (must be spammers)
var spammers = followers.Except(friends); 

And that’s it. Coming up in Part 2, we’ll refactor the code that gets friends and followers to reduce duplication, then we’ll start building out some UI so that we can act on the information we have.

Posted by Jason under General  •  2 Comments