How Our Big Data Platform and Social Analytics API Work

How Our Big Data Platform and Social Analytics API Work Ashu Agte Blogger Extraordinaire Simply Measured

When you use Simply Measured, you interact with metrics, charts, reports, and data tables that help you answer business questions about social media. For example, you might be looking at your conversions and revenue per post on Facebook year-to-date, tracking engagement on Instagram versus competitors, or analyzing post-level data for a spring social media campaign. The big data platform behind the scenes is what makes this all possible.

The end of the rainbow might be a single number, but it’s the machinery behind the scenes that leads to accuracy, scale, and flexibility for you. Here’s how it works.

API MAP

Data Ingestion

Let’s use an analogy here. Imagine that you are gathering water in a big reservoir. Some collection happens by frequent visits to other lakes to bring water back at regular intervals. With some lakes, you can create a stream connecting their reservoir to yours. Similarly, across all the 8+ networks we connect to for data, we follow both of these patterns: we pull data, and data is pushed through to us. The Data Ingestion layer is responsible for managing these connections, organizing data, and recovering missing or bad data. This layer has to be very resilient to network connections failures — an ability to recollect, replay, and gather overlapping details is extremely critical.

The speed and quantity of Simply Measured’s data ingestion is of monstrous proportions, so we decided to build this layer on our own in a unique way (none of the existing technologies were serving our needs). This tier is also responsible for the listening capabilities of our platform. (Tech notes: Highly scalable, fault-resistant homegrown patented tool built in Go and Ruby).

Awesome Things That a Social API Enables

So now we have data coming in from various networks, but it needs to be organized and redistributed to the layers that perform various functions. In simplistic terms, we use a “radio station and radio tuner” model for our distributor. The Data Transmitter layer makes data available like a radio station does for its listeners. There are three listener stations that connect to this Data Transmitter: the Search Cluster, the Report Database, and the Source of Truth Database. The Data Distributor does not persist data and is transient. (Tech notes: This is built using Apache Kafka and Apache Storm).

Multi-Channel Social Media Search Engine

The goal at Simply Measured was to always go beyond metric calculations and measurement into (advanced) analytics, insights, and attribution. In order to get there, we built a very comprehensive cross-channel, enterprise-class search engine.

Metric and Measurement: This search engine now has the ability to provide an interactive web experience as compared to traditional Excel extracts. (Note: We are not moving away from Excel extracts, but adding these features on top as technology and the market evolves and matures).

Basic Analytics and Insights: Search clusters enable search patterns with filters in an interactive manner. You can use keywords and phrases that are not just hashtagged but are also a part of comments across all social channels. We’re baking in complicated metrics like sentiment change by observing patterns.

Predictive Analytics: We have started exploring machine learning and pattern recognition. Very soon we will have the ability to make predictions around campaigns, brand, customer care, and your interactions with consumers and businesses. (Tech notes: We use ElasticSearch to create this massive and highly responsive search cluster.)

Want to know about the rest of our platform and how it gives you the information you need to make better business decisions? Download the guide below for the full explanation. 

Get everything you need to analyze the metrics that matter

How Our Big Data Platform and Social Analytics API Work

Download
Ashu Agte

Ashu Agte

I lead Simply Measured's world-class engineering organization, using the latest trends in technology to build exceptional products and services to the market.