Banjo Leverages Big Data in Real Time

"Everyone is surprised at how well it works, and how little time we spend doing sys admin tasks and can just focus on the product."

Blossom

Powered By

  • 70 Dynos
  • Blossom Database
  • MongoHQ, SendGrid, PubNub, SSL

Requirements

  • effortless back end processing
  • simple scaling for both application & database
  • continuous feature rollout

An App to Tap into the Social Web in Real Time

Pulling geolocated data from one social network’s API into an application can tell you some interesting facts about a place. Pulling and combining geolocated data from numerous social networks tells a more complete story. This story is often not told however. It is difficult to predict how much data one social network will produce at any time, much less five or six. The work of managing such volatility in real time often outweighs the benefits.

Few understand how to solve these issues efficiently like the team building the social networking application Banjo. Banjo pulls every geo-enabled post from the Facebook, Twitter, Foursquare, Gowalla and Instagram APIs into the “Social Verse”. It can then tell you everything that is said in a certain location within all of those networks at that exact moment. Log into Banjo and find out within seconds what people are saying about an event, natural disaster, or just around your neighborhood.

Engineering A Web of Complexity Without the Hassle

Linden Labs alumni and tech veteran Fredrik Björk became Director of Engineering of Banjo shortly after their launch. One of his first moves was to take Banjo off of EC2. “I started using EC2 when it first came out. Over time I tried using Rightscale, Rubber, even Scalr. I discovered Heroku in 2009 and then the question always became, ‘When can we switch to Heroku?’ I don’t want to reinvent the wheel.”

“Everyone was skeptical in the beginning. Heroku has so many customers—why should they care about us?” says Björk. After a week and a half of testing and about a day of migration, Banjo was officially using Heroku. The nine person team could then take their focus completely off of systems administration.

Sudden Spikes in Requests Per Minute Handled Effortlessly

Björk’s decision to switch to Heroku was quickly tested shortly after the migration. After being featured in the Android marketplace and releasing a PR push around new features, Banjo went from zero to hundreds of thousands of users within a month. The influx of users added exponential complexity to Banjo’s web processes within days. “Banjo is like a spider web,” says Björk. “One location can have data from Facebook, Twitter, Foursquare, Instagram and Gowalla. All of that is getting pulled and meshed in real time, every single minute a user accesses the application.” Björk’s team watched as Banjo’s requests per minute spiked from 100 to 1800 within days.

The team simply added ten more dynos to handle the influx of users and 50 more workers to handle the background processing requests from the Twitter, Facebook, Foursquare, and Gowalla APIs. Rather than hiring a database administrator to handle the massive load of records flooding into Banjo, Björk opted to use MongoHQ, a database in the cloud. The team had no issues scaling as more users added complexity to their infrastructure.

Offset Infrastructure Means a Renewed Focus in Features

Since switching to Heroku, Björk and his team push more product updates than ever before. “We are pushing to production several times a week,” says Björk. Now with infrastructure offset to Heroku and MongoHQ, the Banjo team has set their sights on compelling new features and will increase the complexity of their application by adding social networks from both Russia and China. “Everyone is surprised at how well it works, and how little time we spend on doing sys admin tasks and can just focus on the product.”