C# dynamic object and JSON serialization with Json.Net

Standard

I’m working on a project where I don’t need a solid domain model as all the logic is handled by the client-side code using  javascript and various frameworks as SignalR, Knockout.js and jQuery.
That’s why I found myself unwilling to add dumb C# POCO objects just to handle JSON serialization and deserialization in the SignalR hub(s), and I turned to dynamic objects and Json.Net (starting from version 4.0)  to handle this tasks easily and quickly.

Serialization

Let’s say that for example I want to send data to the browser. All I have to do is to create an ExpandoObject (my new favourite .Net class!) and serialize it using Json.Net “JsonConvert.SerializeObject()”.

Let’s see an example:

dynamic foo = new ExpandoObject();
foo.Bar = "something";
string json = Newtonsoft.Json.JsonConvert.SerializeObject(foo);

And here is the output:
“{\”Bar\”:\”something\”}”

Deserialization

Json.Net offers a great way to deserialize a JSON string into a dynamic using the JObject (you can find it under the Newtonsoft.Json.Linq namespace and here the details).

Let’s see an example re-using the previous foo object:

dynamic foo = JObject.Parse(jsonText);
string bar = foo.Bar; // bar = "something"

And that’s it, really easy and yet really powerful.
In a more structured application, I would probably not use this trick as I would probably have all of my ViewModels and Domain Models objects ready to use in type safe serialization/deserialization operations, but that’s not always the case.

I hope this helps.

Valerio

Twitter Real Time Search

Standard

Hello,I’ve always developed web applictions using the ASP.Net Web Forms framework, and so far everything has been ok. A few weeks ago, though, I came up with the idea to re-engineer one of the most important web application we use at work. This web platform runs the business, has been developed using the Web Forms framework and is quite old fashioned: it’s starting to feel a little cramped when we have to add new features and new UI components that should handle complex user interactions with the UI. Not to mention that is not very usable on tablets like the iPad.

Looking for a solution, I thought it would have been great to use the latest available technologies to re-engineer the system… and therefore MVC!

I started learning the ASP.Net MVC framework, along with jQuery, reading the MVC Music Store tutorial .

After that I decided to give it a try implementing a web application on my own, an application that lets the user query the Twitter real time feed.

I had no idea where to start from, I just liked the idea of using the Twitter API!

I’ll start saying that in the following tutorial there’s not much of the true MVC potential about model binding and validation, nevertheless it has been a starting point to deal with such technologies like jQuery, Json, and the foundamentals of MVC.Before we start, I want to point out that this article is not a “Get Started“ tutorial, therefore to all of you that start from scratch I strongly suggest to start reading the MVC Music Store tutorial using the link posted above, otherwise you will miss some key concepts.

Furthermore, I will not post a lot of code here because it’s not really that readable, but at the end of the post you will find a link to download the source code of the project so that you will be able to follow and modify the code as you wish.

Objective

The goal is to develop a web page that lets a single user (to make things simple) see Tweets related to a topic he’s interested in, as they are being created in real time.

Given the single user restriction (later on I’ll explain why), this application is obviously just a toy, nevertheless it’s still useful to cover the basics and could be easily converted into something more useful.

Use case

The user enters a keyword in the search bar and press the “Search” button. After a while, if someone is twitting something related to the user’ search key, tweets will appear inside a box, piling up as they are being created.

When the user is satisfied, he halts the search by clicking the “Stop” button.

What I will use

ASP.Net MVC 3 framework, jQuery and the Twitterizer library (http://www.twitterizer.net/)

Realization

The Twitter developer website is full of interesting and useful stuff if you want to go into more details then the one you will find here.

To start off we must create an application on the Twitter Developer website to start using the Twitter API. Follow the instructions on the dev website and then do as shown in the following screenshot.

How to setup creation of the app

How to setup creation of the app

At the end of the process, you will receive the authorisation keys for the 0Auth authentication protocol.

To handle authentication and web request processes, I’m usingTwitterizer, an ad-hoc library that can be found here.

At the time I’m writing this, current version is 2.3.3, but it’s not correctly working as the address for the stream in the API is changed and is now using the HTTPS protocol. So I had to download the source code, change the address (TwitterStream.cs, row numbers 210 and 212) and import the Twitterizer.Streaming project into my own solution (I liked also the idea of looking at the code).It’s now time to decide how to architecture the project!

Even for simple and quick projects, I like to use a layered architecture, so for this project I created:

  • a Domain Model layer
  • a Data access layer that handles access to the storage to add and retrieve tweets
  • an Application Logic layer that will handle main user operations and access to the Twitterizer service
  • a Presentation layer, built using the MVC framework
What do you say, it’s too much architecture  for such a small project?
Let me show you something :D
The general problem

The general problem

The Domain Model layer

It’s very thin given that the domain can be narrowed down to our representation of a Tweet. The same representation will  also be suitable for the View. Let’s see the class diagram:

Domain Model class diagram

Domain Model class diagram

Following the code for the Tweet class.
namespace Model
{
    public class Tweet
    {
        public decimal Id { get; set; }
        public string Username { get; set; }
        public string ProfileImageURL { get; set; }
        public string Text { get; set; }
        public string Timestamp { get; set; }
        public long? NumberOfFollowers { get; set; }

        /// <summary>
        /// Side-effect free function. Extract URLs, hashtag and mention patterns inside a text and convert them to HTML links
        /// </summary>
        /// <param name="tweetText"></param>
        /// <returns></returns>
        public string FormatTweetText()
        {
            string formattedTweetText = string.Empty;
            formattedTweetText = Regex.Replace(this.Text, @"(http|ftp|https):\/\/[\w\-_]+(\.[\w\-_]+)+([\w\-\.,@?^=%&amp;:/~\+#]*[\w\-\@?^=%&amp;/~\+#])",
               delegate(Match match)
               {
                   return string.Format("<a href=\"{0}\">{0}</a>", match.ToString());
               }, RegexOptions.IgnoreCase);

            formattedTweetText = Regex.Replace(formattedTweetText, @"#+[a-zA-Z0-9]*",
               delegate(Match match)
               {
                   return string.Format("<a href=\"http://twitter.com/#!/search?q={0}\">{1}</a>", match.ToString().Replace("#", "%23"), match.ToString());
               }, RegexOptions.IgnoreCase);

            formattedTweetText = Regex.Replace(formattedTweetText, @"@+[a-zA-Z0-9]*",
               delegate(Match match)
               {
                   string mention = match.ToString().Substring(1);
                   return string.Format("<a href=\"http://twitter.com/#!/{0}\">{1}</a>", mention, match.ToString());
               }, RegexOptions.IgnoreCase);

            return formattedTweetText;
        }
    }
}

The Data Access Layer

This layer is based on the Repository pattern, albeit in a very simplified form, and the repository is created by a factory, using the Abstract Factory pattern.

Let’s first take a look at the class diagram:
Data Access Layer class diagram

Data Access Layer class diagram

The design is pretty easy and straightforward: the concrete repositories instances must implement the ITwitterRepository interface, therefore allowing for an easy swap amongst concrete repositories and no code change is required in the upper layers.
We can retrieve either a list of tweets either a single tweet at a time, using GetRange or GetStatus methods.
Both take a parameter called lastId: this is because we only want to display new tweets and not duplicates.

The concrete repository is created by the RepositoryFactory object, implementing the IRepositoryFactory interface.

In more complex projects, and more susceptible to changes, it would probably be a good idea to use an Inversion of Control tool.

For the purpose of this project I created a concrete repository that works in memory and mimics a queue, and the queue is a static object itself. Thus the single user limitation.
As you can see I didn’ t put a lot of effort in this because it was not really the point where I wanted to focus on. Using a database would be a better idea, or whatever else comes to your mind.

The Application Logic layer

Responsible for handling user generated operations (i.e. start and stop capturing Twitter real time streaming) and subsequent automatically generated requests (more on that later). As shown in the class diagram, there are two object of particular interest, TwitterAccessService and TwitterStreamingListener.

Application Logic Layer class diagram

Application Logic Layer class diagram

The former, as the name implies, implements the Service pattern and is the interface towards the Presentation Layer, the latter is a wrapper for the Twitterizer API. It’s designed as a Singleton (here lies the second cause for  the single user limitation) and is using dependancy injection on the factory method to know which repository to use when a new Tweet or a new event arrives.
Let’s see the code:
namespace ApplicationLogic
{
    public class TwitterStreamingListener
    {
        private static TwitterStreamingListener _instance = null;
        private ITweetsRepository _repository;
        private List<string> _otherEvents;
        private bool _stopSearch;
        private ModelTranslationService _translationService;
        static OAuthTokens tokens = new OAuthTokens()
        {
            ConsumerKey = "yourConsumerKey",
            ConsumerSecret = "yourConsumerSecret",
            AccessToken = "yourAccessToken",
            AccessTokenSecret = "yourAccessTokenSecret"
        };

        public static TwitterStreamingListener GetInstance(ITweetsRepository repository)
        {
            if (_instance == null)
            {
                _instance = new TwitterStreamingListener(repository);
            }
            return _instance;
        }

        private TwitterStreamingListener(ITweetsRepository injected)
        {
            this._stopSearch = false;
            this._repository = injected;
            this._otherEvents = new List<string>();
            this._translationService = new ModelTranslationService();
        }

        public void StartCaptureStreaming(SearchParameters parameters)
        {
            StreamOptions options = new StreamOptions();
            options.Track.Add(parameters.SearchKey);
            this._stopSearch = false;

            TwitterStream stream = new TwitterStream(tokens, "RTSearch (Dev)", options);

            IAsyncResult result = stream.StartPublicStream(
                StreamStopped,
                NewTweet,
                DeletedTweet,
                OtherEvent
            );

            while (!this._stopSearch)
            {

            }

            stream.EndStream(StopReasons.StoppedByRequest, "Stop by user");
        }

        public void StopCaptureStreaming()
        {
            this._stopSearch = true;
        }

        void StreamStopped(StopReasons reason)
        {
            if (reason == StopReasons.StoppedByRequest)
            {
                this._repository.Clear();
                this._stopSearch = true;
            }
            else
            {
                //Do something...
            }
        }

        void NewTweet(TwitterStatus twitterizerStatus)
        {
            Tweet tweet = this._translationService.ConvertToViewModel(twitterizerStatus);
            this._repository.Add(tweet);
        }

        void DeletedTweet(TwitterStreamDeletedEvent e)
        {
            this._repository.Delete(e.Id);
        }

        void OtherEvent(TwitterStreamEvent e)
        {
            this._otherEvents.Add(e.EventType);
        }
    }
}
The static initialization block of code is where you have to put the 0Auth tokens obtained by registering the application on the Twitter Dev site.
In the StartCaptureStreaming method we setup the search key with the following code
StreamOptions options = new StreamOptions();
options.Track.Add(parameters.SearchKey);
TwitterStream stream = new TwitterStream(tokens, "RTSearch (Dev)", options);
And then we start the stream providing callbacks for main events.
IAsyncResult result = stream.StartPublicStream(
                StreamStopped,
                NewTweet,
                DeletedTweet,
                OtherEvent
            );
Whenever a new Tweet arrives, we simply add it to the repository:
void NewTweet(TwitterStatus twitterizerStatus)
{
   Tweet tweet = this._translationService.ConvertToViewModel(twitterizerStatus);
   this._repository.Add(tweet);
}

The repository is injected by the TwitterAccessService object, getting an instance for the repository object by the Factory.Furthermor, TwitterStreamingListener uses the ModelTranslationService object to translate from Twitterizer domain to our application domain model.

These two last objects mimic the behaviour of a (minimal and incomplete) Anti Corruption layer.

The Presentation Layer

This application has only one page to show to the user, and all of the possible actions happens inside it. Therefore the presentation layer will be really thin and simple: 1 page and 1 controller to handle actions.

As I said at the beginning of this article, there will not be much about model binding and validation, a very strong point of the MVC framework.

This is because I chose to use jQuery to load new tweets at runtime.

The page layout is really simple, with a search bar and two buttons to accept user inputs and a box to contain new tweets.

Here is how it looks like:
Search bar and tweet box

Search bar and tweet box

As you can see the page is really simple and clean, with a search bar, two buttons to star and stop the search, and a tweet box to display tweets. Tweets can pile up allowing vertical scrool bar to appear.

The html code is pretty straightforward, so I won’t cover it.

Let’s see the meaning of some of the variables declared in the javascript block of code:

var urlBase = 'http://twitter.com/';
var serviceStarted = false;
var timeOut = 0;
var singleTweetMode = false;
var last = 0;

“serviceStarted” is used to signal the javascript code that the user started a search.

“singleTweetMode” can be set to true to mean that we will display inside the tweets box one new tweet at a time, viceversa we will display several new tweets at a time, therefore pushing down all the old ones.

“last” is used to store the Id of the last tweet that has been shown, so that tweets will be shown respecting their original timeline.

Now let’s see the code in detail based on the different actions performed by the user.

Search

after pressing the Search button, the control passes to the javascript run() function, which, after validating user input, signal to the page that the service is started via the “serviceStarted = true” line of code.

Then, based on the operational mode that we chose via the singleTweetMode, tells the javascript engine that every x milliseconds it should invoke the poll() function.

After that, the application performs an Ajax postback with jQuery, passing the search key as a parameter for the HomeController hosted StartSearch action method, that will then ask the TwitterAccessService to start searching for tweets based on the keyword

function run() {
        if (!serviceStarted) {
            var value = $('input:text[name=searchKey]').val();
            if (value == null || value == '') {
                alert("Please enter a search keyword!");
                return false;
            }
            $('#SearchButton').val('Searching');
            serviceStarted = true;
            if (singleTweetMode) {
                timeOut = window.setInterval(poll, 2000);
            }
            else {
                timeOut = window.setInterval(poll, 10000);
            }
            //$('#SearchButton').attr('disabled', 'true');
            $.post('/Home/StartSearch', { searchKey: value });
        }
}
The HomeController will then invoke the service and then the TwitterStreamingListener object’ StartCaptureStreaming method.
The javascript poll() function therefore performs an Ajax postback to the GetTweets action method (hosted by the HomeController), using function displayNewTweets as a callback to invoke when data has been returned.
$('#LoadingImage').css("display", "inline");
$.post('/Home/GetTweets', { lastId: last }, displayNewTweets);
Let’s now see the code for the GetTweets action method
//
// Ajax: /Home/GetTweets
[HttpPost]
public ActionResult GetTweets(decimal lastId)
{
     JsonResult serializedTweets;
     if (singleTweetMode)
     {
         if (ViewBag.Count == null)
         {
              ViewBag.Count = 0;
         }
         else
         {
              ViewBag.Count++;
         }
         var tweet = service.GetTweet(lastId);
         serializedTweets = Json(tweet);
      }
      else
      {
         if (ViewBag.Count == null)
         {
             ViewBag.Count = 0;
         }
         else
         {
              ViewBag.Count++;
         }
         var tweets = service.GetTweets(lastId);
         serializedTweets = Json(tweets);
       }
       return serializedTweets;
 }
It’s very simple: it invokes the right service method based on the mode (a variable present also in the HomeController) whether it’s for single tweet or several at once, and then serializes the result in Json notation to give the control back to the callback javascript function displayNewTweets.

I serialized data in Json format in order to return data to javascript in a way that it’s simple for it to manage and natively supported by the framework.

Type JsonResult is of type ActionResult, so the method’ signature is still valid.

Upon returning from Post, the displayNewTweets javascript function is invoked, creating the necessary HTML code to display retrieved tweets:
function displayNewTweets(data) {
        $.each(data, function (count, item) {
            if (last < item.Id) {
                last = item.Id;
                addNew(item);
            }
        });
        $('#LoadingImage').css("display", "none");
}

function addNew(item) {
   $('#TweetsBody').prepend(renderTweet(item, 'hidden'));
}

function renderTweet(item) {
        importanceColor = getImportanceColor(item.NumberOfFollowers);
        return '<div class="tweet" id="' + item.ID + '">' +
               '<strong><a href="' + urlBase + item.Username + '" style="color:' + importanceColor + '">' +
               '<img src="' + item.ProfileImageURL + '" class="ProfileImg" alt="Usr image" />' +
               item.Username + '</a></strong><span class="text">' + ' ' + item.Text +
               '</span><span class="created_at"><br /><a href="' + urlBase + '#!/' + item.Username + '/status/' + item.ID + '">' +
               item.Timestamp.toString() + '</span></div>';
}

function getImportanceColor(number) {
        rgb = 255 - Math.floor(16 * (Math.log(number + 1) + 1));
        return 'rgb(' + rgb + ', 0, 0)';
}
In the addNew function the new tweet is put on top of the list, using the “prepend” built-in function.
Function “renderTweet” is just a matter of building up the HTML code to display the new tweet. The username is colored based on the importance of the user, and the importance is based on the number of followers.

Stop search

When the user presses the Stop button, the javascript stop function is invoked

function stop() {
        if (serviceStarted) {
            $('#SearchButton').val('Search');
            $('#StopButton').val('Stopping');
            window.clearInterval(timeOut);
            $.post('/Home/StopSearch');
            $('#StopButton').val('Stop');
            //$('#SearchButton').attr('disabled', 'false');
        }
}

The if statement prevents the user to perform an actual Stop operation even though no search is currently running.

After the check I need to clear the timer that is causing the poll function to be invoked over and over

window.clearInterval(timeOut);

Then I need to perform a post to notify the Twitterizer lib to close the stream and release the resources. To make this operation easy I chose to use the Singleton pattern so that only one stream can be open at once, and therefore it’s easy to retrieve its reference.

References

To write this tutorial I actually used and got inspired by a several web tutorial, stackoverflow posts and books. Some of them are:

The source code can be downloaded here .

That’s it, I really hope you enjoyed this article!

Valerio