DJ Aoki playing a special set and interacting with the audience on Turntable.fm

As I jumped from room to room on Turntable.fm last night my eyes caught a glimpse of a rare room titled “AOKIxSOLREPUBLIC” . I clicked it with a fury. “Sorry due to fire codes you have been escorted out of the building” was the message I received, which was Turntable’s cute way of telling me that the room was already full. I reloaded the page. Same message. For the next 5 to 10 minutes I kept refreshing the page until I finally secured my seat. I was in.  I had one of 200 spots available to hear the famed DJ Aoki play some of his favorite tracks.  As I sat there listening to the music, I couldn’t help but think this was a monumental shift in the way music is being consumed online.  And even more incredibly, the value being delivered could potentially be worth billions to an industry used to losing atleast that much.

I know, I know. This story has been told many times before.  There have been countless music startups that have been birthed (and subsequently deadpooled) over the years. Even so I stand by my claim, but first you’ll need a little bit more context.

1) It was unthinkable that we had over 1 million people using our service and
 
2) It was unthinkable that we were actually still in business (we had survived lawsuits, and many employees – including myself – had worked without pay for 6-10 months).

Like this article?  Want to receive updates on data sets I create, other articles, or updates from the community?  Subscribe to the mailing list!

Have you ever thumbed through the Games section of the App Store and wondered how some games get to the Top 25–and stay there?  Or if there were any common elements of gameplay that resonate well with certain groups of users?  Or perhaps you’re a developer who’s interested in IPhone games and you’ve observed some anecdotal clues but have had trouble finding data to support your intuition?  Do you want to make more money with your existing IPhone game?

I recently spent some time looking at a lot of Free-To-Play Games data, mostly through App Data and App Annie, as well as played somewhere around 84 different titles from the top games studios. This data is freely accessible but a pain to cobble together.  Why would I do such a thing…you might ask? Well, partly because I love data and love uncovering something interesting from disparate data sets.  Partly because I make my own IPhone apps and am trying to figure out how to navigate the App Store’s competitive landscape.  And partly because I work at GREE as a data analyst and want to fully understand the type of product GREE is bringing to the market, and its potential users.  I also happen to be a gamer myself!! (SC2, Angry Birds, Halo, and MinoMonsters)

Compiling all this market research took quite a bit of manual labor, but I learned a few things about this competitive market and thought it would be beneficial to my readers as well.  Many of you are developers and might find these types of insights valuable, not only for developing games, but other applications as well.  So here you go!  My Forbes-like list of Top  5 Tips to Make your Game Successful:

Tip #1:  If you want to make money, target women

The data overwhelmingly supports the notion that women are the biggest spenders in the Top Grossing Ranks category of app store Games section.  At the risk of appearing sexist, I classified all 84 titles as either male or female depending on what I thought was a best-guess target market.  For example, I made the assumption that the games Top Girl and Social Girl by Crowdstar are game titles targeted to women, while Blood and Glory a title by Glu Mobile would be targeted to men.  And if there were games that I just felt too conflicted to make the call, ie Angry Birds, then I labeled it as “Either”.  The result?  46 games were targeted to women, 34 to men, and 4 either.  The average top grossing rank of a female-targeted game was 113 vs 145 for men.  Revenue is tied to rank, and it grows rather exponentially.  From sources that I’ve gathered from around the web, most notably Inside Mobile Apps, female-targeted games bring in almost $100k more per a month, on average, across the top 100 ranked games.

Tip #2:  Offer a grab-bag of virtual currency as an In App Download

While I’m still tabulating numbers, I noticed that most (meaning possibly > 70%) of the Top Grossing games offered virtual “currency” as an In App Download.  I use the word currency as a broad term here.  For some games like Cityville by Zynga, the IAP’s are quite literally USD for Zynga Dollars, or whatever their equivalent name is.  Probably something like “Bag O’ Dolla’s”.  Other games trade “energy” for money, meaning if you want certain events to go faster, pay more money.  I can think of a few reasons why game designers do this: a) It makes the most sense for users, so there is less friction during the buying process. b) Its easy for game developers because Apple’s IAP process is a royal pain in the ass to implement and c) Its easy to scale for whales (see Tip #3).

Tip #3:  Plan BIG for Whales

It’s no secret that there is this crazy phenomenon within games that your most loyal users will spend ridiculous amounts of cash to improve their gameplay experience.  This will stun you but has been vetted by Zynga’s S-1 filing.   As much as 25-50% of Zynga’s revenues come from these 1% of users.  Even articles written about it.  These ballin’ users are dubbed “whales” and you need to absolutely, without a doubt, target/incorporate/cater/suck-up to them in your game design. They will reward you handsomely.  So how do you plan big??  Take whatever “currency” you’ve decided for your game, and offer a few top tiers around the $50 to $100 per purchase mark.  Simple as that.

Tip #4:  You will probably have to buy users, but you won’t know what good it’ll do

Not much of a tip, I know.  But in this competitive landscape, you’ll have to atleast start out by buying users through a network such as AdMob, InMobi, or TapJoy.  Your main objective is to boost the ranking of your app as high as your budget will allow.  Remember, revenue scales nearly exponentially with App Store ranking, atleast with the Top Grossing Games category.  The environment right now sucks though because unlike with Google Ad’s for Web, your conversion funnel will be crippled by Apple’s policies and distribution.  For example, tying a download to a CPC creative is problematic ever since Apple announced the deprecation of the UDID.  Some networks show download conversions, some do not.  It will be your responsibility to A/B test and figure out if the value of your users are worth more than the ad-spend.  Good news is that clicks can be bought for as low as $0.04-$0.08.  You just won’t know whether these users actually spent the most money on your game or referred others who did.

Tip #5: Push, Push, Push, and Push some more

Push notifications are a remarkable retention mechanism on the IPhone.  But use them intelligently because they can create an annoyance for users or can be quite embarrassing.  During my testing (I swear this was testing!) I was repeatedly pinged about the status of my date on Social Girl or that my teepee reward was ready on Alien Family.  Since push notifications give the same sound (often, but sometimes not) and the same vibration as an SMS, I was often fooled to open up the notification during a time that wasn’t entirely ideal. Like drinking with my friends at a bar. Ouch.  With that said, because these notifications are so uniform between applications, its a viral channel that you must utilize. And surprisingly, many popular apps do not, like Tetris and Angry Birds.

I hope I fused a little bit of data and a little bit of anecdotal experience to shed some light on what you can expect when developing an application for Apple’s App Store, particularly a Free to Play game.  While not entirely conclusive, I will continue to firm up these numbers and offer more concrete resources (possibly some open-source datasets!) for the community.  If you’d like to be notified when these data sets become available, please sign up for my mailing list!

http://kaleidoscope.kontagent.com/2011/11/09/big-data-is-useless-without-science/

Many years ago I researched explosives by shining a light on them. It was every bit as exciting as it sounds. We would shine a light, take a picture, then study the explosive to see if it changed. I would painstakingly scour thousands of data points, looking for small fluctuations in intensity, signs of discoloration, or any statistically significant feature. We collected immense amounts of data from sensors, but the explosive always looked the same when we took snapshots. Then eventually we found out that if we looked not just at the snapshots, but also at the differences between the snapshots using a mathematical formula, we could see dramatic changes. We found out that every explosive was different, and we could effectively detect an explosive from a distance by just shining a light. Today, that research is being used to scan people before they enter airports for bombs.

Today, companies have more customer data than they can handle. Like a digital version of the show Hoarders, companies try to keep every bit of detail for as long as possible with the hope that one day these useless bits can be turned into massive new revenue opportunities. Over the past five years, bright engineers have devised open-sourced solutions to store and process the data deluge. We now even have a “big data stack” — that is, a framework for commoditizing data.

A simplified big data framework

 
At a previous job, I was essentially the housekeeper of our home-built data infrastructure based on Hadoop. The idea was to create a monolithic tracking system that would record all anonymous user data, batch it up and store it in a distributed store. We’d then process and query it to find fascinating facts about our users that would help drive product roadmaps. At least that was the idea. Sound familiar? In practice though, we weren’t exactly sure what to track. Which user attributes were meaningful? How would we identify statistically significant behavior? Were we sure of data accuracy enough to influence the product roadmap? It turns out that we–and many other companies it seems–neglect a major component of the “big data stack”: science. So let’s modify our framework.

An actionable big data framework

This diagram may seem a bit strange if you’re a software engineer. Because typically when you talk to engineers about big data, you’ll hear a litany of tool sets that sound like characters out of a Harry Potter novel: Voldemort, Pig, HDFS, Oozie, Zookeeper, Flume, Hive, Cassandra… you get the picture. We have yet to get to a point where science can be commoditized, and perhaps it never will (though Mahout is a step in the right direction). Despite all these tool sets, scientists will always be needed for their intuition, interpretation, and curiosity. Scientists are needed to analyze the business needs of a customer and ask the right questions to solve critical business problems. Scientists are needed to transform an ugly piece of log data into a beautiful infographic that can spur an organization to launch a new product, bolster existing services, or otherwise remain nimble in a highly competitive economic environment.

These scientists are not scientists in the traditional sense. Their domain isn’t necessarily their bachelor’s, master’s, or dissertation topic. On the surface, my background in explosive materials doesn’t sound like it helps me with my day job of helping clients understand user behavior in their mobile and social applications. Instead, I’ve learned that “data scientists,” no matter what their background, specialize in providing insight by using keen analytical and quantitative skills. If needed, they will clean, explore, and model data sets to create new information products and key metrics. These scientists are not in a cubical doing mundane research towards an elusive goal. They are highly collaborative and “high-touch,” that is, they constantly communicate with a key stakeholder so an end goal is reached.

The scientific process and research I conducted on explosives was instrumental in creating a product that went to market. The organization I was a part of taught me to be curious and to look at data sets in new ways. We had the infrastructure to store, process, and query the data, but ultimately it was our insight that produced a working prototype. In today’s data-driven world, the organizations that best leverage their data and invest in the right people to derive insights from that data will gain large competitive advantages over organizations that fly blind.

The Question:  Do Southern states really listen to more country music?

All good experiments start with a question.  Before every project, I come up with a question I want to answer so I do not lose focus as the work gets more complicated.  For this project I wanted to pick something insightful and relatable.  An excellent example of a model I wanted to follow is Nathan Yau’s U.S Unemployment Map.  This is ultimately how I wanted to display the answer to the country music question.  I envisioned in my head that Louisiana, Mississippi, Georgia, and Florida would all be dark-colored and the rest of the United States would be lighter shades of that same color.  Assuming that color represented country music popularity, that would mean that the answer to my question was “YES”!  I also envisioned that if I got access to a dataset that was timestamped and collected data for a few weeks, that maybe I’d be able to put together a fancy little movie showing the transition of certain genres over time.  Seeing this visualization could perhaps point to macro trends like the distribution of a new single by a popular artist, a concert that was making its way across the U.S, and of course the affinity certain counties had for certain genres of music.

The Twitter #NowPlaying Dataset

The next step was to get the data.  Data requirements for this experiment were: recency (semi-important), free (important), large (important), location-based (important), and parse-able (semi-important).  I immediately thought of Twitter as an appropriate data supply.  Twitter has an excellent push API that constantly sends new status updates that contain all the important bits about a Tweet you can think of: urls, hashtags, mentions, timestamps, and location. All for free. Requirements, check!  Now, I didn’t really want just any tweet, I wanted tweets just about music.  As a casual Twitter user, I’ve often seen how people will use the #nowplaying hashtag to say what they’re currently listening to on their streaming music platform of choice.  So I decided to filter the API looking only for the #nowplaying hashtag.  Data started to flow in like Niagara, so I wrote a little program to catch the tweets, batch them up, and dump them to disk.  I then promptly forgot about that process and came back a couple weeks later.

Problems

When I began to sift through the data to put my visualization together, I began to realize that the mountainous amounts of data I was expecting turned out to be really not that much.  The raw data was quite large, but the final form of the data I needed to make the visualization work was <FIPS Code, Value>  where FIPS is the Federal Information Processing Standard for plotting county-level data and value was the popularity value for genre in that region.  I thought I’d use raw tallies of the genre for that FIPS code to start off.  However, Twitter doesn’t give location data in FIPS.  In fact, Twitter gives what users end up writing in the location field, so that could be anywhere from the nicely labeled “San Francisco, CA” to the absurd but Twitter-normal “Somewhere in the middle of nowhere”.  After filtering out all these “bad” locations, I was left with a fraction of the raw data.

The next step was to translate the #nowplaying data to genre data.  For example, a typical entry might look like “#nowplaying Lady Gaga – Monster”.  To translate the Artist/ Song combo to Genre, I grabbed a Discogs dataset with Artist to Genre columns and Song to Genre columns, then matched them to terms within the nowplaying string.  Of course, after this string matching, many entries were eliminated because they either a) weren’t correctly formatted as Artist – Song or Song- Artist b) Didn’t have Discogs data or c) Discogs data didn’t have any genre data.  After all these filters and matching and distillation, I then reduced the set to FIPS, Value pairs where I could then plot them with the convenient D3.js library choropleth.  The result?  Well, not exactly what I had envisioned

  U.S Choropleth Thematic Map Fail

Lessons Learned

After all was said and done, I did learn some important lessons.

1.  U.S Thematic Maps requires loads of data.  While you don’t necessarily need to visualize every county in the U.S, to show a relative majority of the counties you need a very detailed dataset.  If you scrape the data from most popular services (ie. Twitter) you’re going to be bound by Twitter’s own popularity in specific areas of the U.S (or elsewhere).  The reason why visualizing U.S unemployment works but music genres doesn’t is that the U.S Department of Labor keeps very accurate data for every county in the U.S.  So choropleth maps might make sense for other similarly detailed datasets.

2.  Zipcodes > FIPS??? When you visualize data, you have to munge the data to be portrayed in the right units.  The aforementioned U.S choropleth map takes in a key-value pair of FIPS code to magnitude value.  That didn’t work for this experiment.  But there are plenty of other options.  For example, I could have used Zipcode instead and plotted the data using Ben Fry’s technique.  Or if I really wanted to be fancy, I could have made my own custom boundaries from GPS data and translated them into shapefiles which could then be filled in.  But that’s perhaps another tutorial….

3. Take into account dataset filtering.  I’ll be honest, I completely underestimated how much data I would have to discard.  At every point along the pipeline of shuffling data from one stage to another, I seemed to have lost 50% volume.  I failed to account for how much this would affect my final results, especially after I reduced the raw data to its “plot-able” form.

Hope this breakdown was beneficial. In the future I might put together the code samples I used to conduct this analysis. Until then…time to start planning the next project!

For the past few weeks, I’ve been working on a microISV (Independent Software Vendor) business called FitLabs.  The goal of FitLabs is to measure your fitness.  I’ve always been a workout fanatic. I’ve done sports all my life from Cross-country to Track and Field, Crew, Soccer, Basketball, and now CrossFit.  When I joined CrossFit, I realized real quickly how data-intensive it was. I was given a binder and every day I wrote down the workout, the times, the reps, and the weights.  But after a while, I really didn’t know how to make the best use of all my journaling and data.  Its now been a year, and I’ve grown tremendously.  I feel stronger and healthier and more flexible.  I’m also happier, but that may be because I just moved.  But despite all this progress I still have some burning questions about my fitness and health like,

• Am I really stronger? How has my weightlifting gotten better?
• If I attempt to lift more weight, can I handle it?  Am I at a local or global minimum?
• Am I completing my workouts faster? Am I doing more reps than I used to?
• How is my performance compared to other athletes of my height and weight?
• How has my diet influenced my health?  Change I change my diet to have a quantitative impact (increase) in my health?

I decided to build an application from scratch that could begin to answer these questions.  It’s nowhere near perfect, but its starting to illustrate some answers.  For example, after loading in my workout journal, I was able to visualize my Front Squat

FitLabs was able to show me how much weight I have been able to move in my front squat lift.  Its leaves a little to be desired like how many reps I did and how much barbell weight I was pulling, but I will improve on that.  A little further down the page, I show some basic statistics like,

This shows me (ideally) the range of my front squat lifts.  I’m able to see what my max working weight was in a workout (note, I have excluded strength workouts such as 1-rep maxes), and that record was set back in November.  One month later, I used a drastically lighter weight, which could have been due to a variety of factors.  The point being, when I’m working out, I don’t know what these statistics are.  Now when I go to the gym, the visualization is clearly in my head.

FitLabs has some bugs, but I am dedicated to perfecting it because I’m betting on the fact that if these questions are important to me, they must be important for others.  If you find this interesting, and want to check it out (free of charge) please contact me.  Otherwise you can signup for FitLabs.

I’ll be posting about my experiences getting this software business off the ground. I’m a complete noobie, so I’m looking forward to hearing from the community!

Digital Music News has an interesting visualization of the changing climate of music sales.  The statistics are astounding (yet not really surprising) to see. In 2002, CD sales were the overwhelming majority of music sales at 95.5% market share.  Last year, they made up just 49.1%, having been eclipsed by the Digital Single (I-Tunes).  Its clear why the record industry is clutching to their dying business model — gone are the times where a customer would spend up to $20 per a CD.  Now people are much more selective about what music they buy, and the Digital Single presents an opportunity to the consumer to be more choosy.  I bet if we look at a graph of music industry revenue over the years, we will see some serious declines.

Ask the people behind any popular web service and they will tell you that a large portion of their user base is located outside of the US.  Web applications are increasingly being made with global audiences in mind, and therefore a need has arisen to represent that geospatial data.  Visualizing geographic data can be a challenge, even when focusing on just the United States.  Shapes need to found and downloaded, coordinates need to be generated, then the appropriate data must be mapped to it and plotted in a form browsers can recognize.

Luckily, the tools and databases for plotting geographic data have gotten much better.  The US Census keeps an accurate record of boundary data, which you can the plug into MapShaper to create a polygon mapping.  There are also maps freely available at OpenStreetMap and Google Map API.  You might also find it beneficial to great a GeoJSON file instead of using shapefiles, so they can be used in libraries such as Polymaps.  Use ogr2ogr. An example of usage is here.

Using SVG to represent the data presents a clear advantage as you can manipulate the data and its design using simple CSS rules.  The Polymaps and D3 libraries are excellent for visualizations that can be drawn directly onto the DOM.  They render very quickly.  You can also pre-process an SVG using Beautiful Soup and some Wikimedia templates.  FlowingData has an awesome tutorial for creating a beautiful visualization.

If SVG is not your cup of tea, you can also use Flash.  Another library suite with lots of features is AmMaps.

I’m currently working on a project which will hopefully pull some of these concepts together in an easy-to-use tutorial.  If you’d like to be notified when it goes live, feel free to sign up for the newsletter below.

UPDATE:  Well, so much for wishful thinking.  I completed the project, but it didn’t quite go as planned…

If you’re sick of your morning and evening commutes, you may be wondering how close you can move to your work while living affordably.  Of course, there are many other factors that go into choosing a place to live, but what if you can see the relationship between home price and transit time?  One Bay Area is a website dedicated to just that. Created by a consortium of agencies, the effort is meant to create a more sustainable future by planning for future land use and transportation.

Don’t know anything about ngrams?  Ngrams are essentially word/phrase counts.  They are used extensively in speech recognition in training Markov Models.  Anyways, Google’s Ngram Viewer is simple and effective and can unlock some interesting trends such as the couple I discovered below:

The Feminine Mystique

Technological Revolution

Sample Data (Lines)

A whiteboard – I say this somewhat jokingly.  But the truth is that a simple whiteboard sketch of a sample set of the data will help you brainstorm important questions to ask, or uncover complexities of the data you may need to plan out before writing any code.

Prototype Data (KB – Low MB)

Analysis and Visualization:

Matlab – Proprietary software that most universities will have licenses to or allow students to buy cheap copies.  Matlab is basically an IDE with great support for visualizing data in native structures (arrays, matrices, cells).  This comes in handy when working with the many, many wonderful academic toolboxes for such things like Neural Networks, Machine Learning, and Machine Vision.

Octave – Open source implementation of Matlab.  Great for when you eventually have to leave the nest of the University but are still hooked on Matlab.  Most Matlab toolboxes are supported and many of the original functions are preserved.

R – Open source statistical computing environment.  I didn’t come from a statistics background but I’m curious to learn it.  Used by many professionals and academics in the field.  See FlowingData for more practical illustrations.

Processing – Visualization development environment.  Creates beautiful visualizations and comes with an IDE.  Check out Ben Fry’s Visualizing Data.

Online Data (MB – Low GB)

This is the point where handling data becomes a bit unwieldy.  Perhaps this is when you’d like to provide analytics as a web service, and you’re not prototyping anymore on your local workstation.  So assuming you have to start migrating your experiments to production deployments on a server you need to start separating the components needed to do the analysis and visualization.

Storage:

MYSQL (or other DB) – Open source and used by many companies to query and serve data.  Write SQL statements to mine data.  MYSQL has support for some statistical functions but performance may degrade if the data size is near upper bound.

Analysis:

NumPy and SciPy (Python) – Because of its roots in the scientific community, Python has great support for mathematical functions that may be absent in other languages such as PHP.

Weka – Free Java-based software for data analysis and machine learning and visualization.  Has database connectivity.

BLAS/LAPACK – Core Linear Algebra Fortran packages which NumPy and Matlab are based off.  Use the C API to write your own custom wrapper in any language that suits you.

Visualization:

Flare (Flash) – Build animations with Actionscript using the Prefuse toolkit. Supports rich features in data modeling and contains animations for trees, graphs, and tables.

HighCharts (Javascript) – Easily construct standard pie, bar, and line charts using JQuery.

AmCharts (Flash) – Construct pie, bar, and line charts. Also has map support.

Protovis (Javascript) [Development stopped for D3.js] – Creates SVG images that can be inserted into page. Great for creating custom visualizations that might become tedious with other frameworks.

D3.js (Javascript) – Built directly from the standards of CSS3, HTML5, and SVG and does not require learning a new graphical representation, like Protovis. Operate on the DOM directly and apply data-driven transformations.

Google Visualization API – Built from HTML5, SVG, and Google Web Toolkit.  Google makes it easy if you want to develop your own Google Trends by supporting many basic types of visualizations like scatter plots, bar charts, and Google Maps integration.

Raphael.js (Javascript) – A JS library that makes it easy to create vector graphics using SVG and VML (deprecated).

“Big” Data (GB – TB – PB)

At this stage, data cannot be stored on a single hard disk, so it must be distributed among many machines.  This is an active area of development for large websites, and so far the open source community has adopted Apache Hadoop as the flagship software framework for handling big data problems.

Storage:

Hadoop Distributed Filesystem (HDFS) – Creates replicas of data blocks and distributes them on compute nodes in a cluster.

HBase – The Hadoop database.  It provides random, realtime read/write access to HDFS.

Analysis:

Hive – The data warehouse system that can be used for offline batch processing, ad-hoc querying, and statistical analysis of large datasets.  Hive uses a SQL-like syntax that makes it easy to rapidly develop queries.

Mahout – A framework for deploying many machine learning algorithms on large datasets.  The algorithms have been rewritten with scale in mind.  Currently the main use cases include: recommendations, clustering, classification, and text mining.

Visualization:

There is not yet a popular framework that can view the distributed data in its raw form. If you know of one, please leave the suggestion in the comments or contact me.  Typically the industry practice is to summarize the data and collapse the feature-set, then load it into a database such as MYSQL for online querying.  From there you can use any of the aforementioned visualization software libraries.