Big Data and the Goldilocks Principle

I was inspired to write this post (I can hear all of you sighing ‘Yet Another on Big Data’) due to another ‘Big’ reason. I listened to a TED (www.ted.com) talk by David Christian titled ‘The History of the World in 18 minutes’ in which he narrates a complete history of the universe, from the Big Bang to the Internet, in a riveting 18 minutes. This is “Big History”: an enlightening, wide-angle look at complexity, life and humanity, set against our slim share of the cosmic timeline. Check out his website – www.bighistoryproject.com , and I promise you that this ‘Big’ has nothing to do with Big Data, as we know it. But what got me interested in his talk is his reference to the ‘Goldilocks moment’ – a moment so precisely right for certain thresholds to be reached to enable higher forms of complexity (life) in the universe.

That got me thinking – Is Big Data the ‘Goldilocks moment’ for organizations with respect to analytics helping them towards achieving better business outcomes?

I think the answer is ‘Yes’ and this stems from the following hypothesis – An organization can utilize analytics for better business outcomes if:

a)      they have more data points to be analyzed (volume)

b)      have the ability to perform sophisticated analysis on large and diverse datasets (variety)

c)       and can do it at a much faster rate than before (velocity)

In that context, I really liked the picture (given below) from one of the IBM articles, which illustrates how Big Data when synthesized  properly along with standard transactional data can help in better business decision making (in this case, it was Fraud Detection)

Source: IBM – Understanding Big Data by Paul Zikopoulos

On the other hand, the exponential increase in processing power of CPUs, the steep fall in memory prices and high bandwidth availability, have enabled the practical use of Big Data techniques. From the human angle, people are creating digital data, viz. social media chatter, video sharing, blogs, mobility etc. at a rapid pace that organizations (with help of Big Data techniques, of course) can potentially solve the ‘Innovators Dilemma’ by providing new products and services that the consumers did not ask for simply because they couldn’t figure out what they actually want.

All in all, I think we are at a precise moment in history (the Goldilocks moment) where organizations can greatly increase their ability to provide better products & services for their consumers using Big Data techniques.

Source: http://blogs.hexaware.com/business-intelligence/big-data-and-the-goldilocks-principle/

Read more »

Business Focused Analytics – The Starting Point

Having been a Business Intelligence practitioner for the last 13 years, there has never been a more exciting time to practice this art, as organizations increasingly realize that a well implemented BI & Analytics system can provide great competitive advantage for them. This leads us to the question of – ‘What is a well implemented BI system?’ Let us follow the Q&A below.

Q: What is a well implemented BI system?

A: A well implemented BI system is one that is completely business focused.

Q: Well, that doesn’t make it any easier. How can we have BI that is completely business focused?

A: BI & Analytics becomes completely business focused when they have ‘business decisions’ as the cornerstone of their implementation. The starting point to build / re-engineer a BI system is to identify the business decisions taken by business stakeholders in their sphere of operations. Business decisions can be operational in nature (taken on a daily basis) and/or strategic (taken more infrequently but they tend to have a longer term impact). To reiterate, the starting point for BI is to catalog the business decisions taken by business stakeholders and collect the artifacts that are currently used to take those decisions.

Q: The starting point is fine – What are the other pieces?

A: The next step is to identify the metrics and key performance indicators that support decision making. In other words, any metric identified should be unambiguously correlated to the decision taken with the help of that metric and by whom. Next we need to identify the core datasets in the organization. Please refer to my earlier blog post titled ‘Thinking by Datasets’  on this subject.

Q: What about the operational systems in the landscape? Aren’t they important?

A: Once we have documented the relationship between Business Decisions to Metrics to Datasets, we need to focus on the transactional applications. The key focus items are:

• Inventory of all Transactional Applications
• Identify the business process catered by these applications
• Identify the datasets generated as part of each of business process
• Next step is to drill-down into individual entities that make up each of the datasets
• Once the Facts & Dimensions are identified from the entities, sketch out the classic ‘Bus Matrix’ which would form the basis for dimensional data modeling

 

Q: All this is good if we are building a BI system from scratch – How about existing BI systems?

A: For existing BI applications, the above mentioned process could be carried out as a health-check on the BI landscape. The bottomline is that every single report / dashboard / any other analytical component should have traceability into the metrics shown which should then link to the decisions taken by business users. BI & Analytics exist to help organizations take better business decisions and that defines its purpose & role in an enterprise IT landscape.

The answers mentioned above provide the high-level view of Hexaware’s approach to Business Intelligence projects. We have worked with many organizations across industries and a business focused analytical approach has provided good value for our customers.

Thanks for reading. Please do share your thoughts.

Read more »