Business Intelligence Vision

Vision for the new BI 2.0

la·ten·cy (l_t’n-s_)n. “ The period of apparent inactivity between the time the stimulus is presented and the moment a response occurs.”

Almost every business practice has adapted to shortening business cycles, except for the Business Intelligence world. Why do we run our business operations disconnected from the insights that could make

us more money and reduce costs? in an ‘always on world’ where we run our lives minute by minute, wirelessly connected to a wide variety of information sources, why should we run our businesses based on out of date information? Clearly something has to change in the way that we (human beings) use technology to process and analyze data.

So what does BI 2.0 change and why?

We discussed how latency has such a dramatic effect on BI processes. New BI 2.0 architectures transform the way that we access data, analyze data and drive decisions.

The goal of BI 2.0 is to reduce all three of these latencies to milliseconds, effectively zero in practical terms, and therefore to maximize value. Today this value is often an opportunity missed, or an expense, or a risk that has to grow large enough to be spotted manually.

Data latency can be reduced dramatically by using in-memory processing in place of storing data on a disk. Analysis latency can be reduced by automating the interpretation of the data, so that people do not need to look at every item, only the problems. In order to do this, of course, you need to be able to understand which business events are problems or could become future problems. And finally, decision latency can be eliminated in many operational (and tactical) decisions by automating the actions.

So in a BI 2.0 world, as stuff happens in your every day business, the BI system can automatically store the data in memory, analyze and interpret the event’s significance, and in many cases, initiate a corrective action without any human involvement. And BI 2.0’s ability to do each of these steps in real time is integral to any automated corrective action, or to the generation of an exception notification. In short, your business can become more intelligent, as processes become smarter.

This is often referred to as a ‘closed loop’ and while the idea is not new, existing BI architectures based on data warehouses don’t fare well.

Indeed, many people haven’t made the connection between real time and closed loop – without real time, the value of an automated action diminishes rapidly. For example in the CRM world, it’s well known that a real time, highly targeted response to a specific customer event will achieve a 50% increase in response rate compared with an offer made days later.

“ So in a BI 2.0 world, as stuff happens in your every day business, the BI system can automatically store the data in memory, analyze and interpret the event’s significance, and in many cases, initiate a corrective action without any human involvement.”

So in order to deliver BI 2.0, very different architectures are required from the traditional data warehouse centric ‘extract-transform-load-query-analyze’ approach of today. BI 2.0 architectures are event based – i.e. these systems analyze streams of event data, not static data stored in data warehouses.

Many processes, of course, cannot be modeled easily, and cannot be automated. For example, in security applications, human intervention is essential. If a member of staff needs training, disciplining or dismissing as a result of their actions, this must be done in person.

What is needed under these circumstances is to detect the problem as rapidly as possible, and present the investigator with a concise alert, containing all of the facts needed to support the decision. It’s critical that these alerts make the human decision process efficient – everything needed to make the decision should be contained in the alert itself. Today, much of day-to-day operations exception handling is characterized by reference to multiple operational data sources and reference reports to supplement and cross check the information.

While this may be the goal, in many businesses today the critical event remains elusive due to the volumes of data involved.

Event Driven BI

BI 2.0 is fundamentally different to traditional Business Intelligence. Data is not stored in a database; queries are not run against the database; data is not extracted for analysis. BI 2.0 uses Event Stream Processing.

It’s worth just stopping to digest that.

We have become so accustomed to all BI being based on queries running against a data warehouse or some other underlying database, that this new BI 2.0 architecture needs considering carefully. After all, this is a radical departure. Event Stream Processing, as the name implies, processes streams of events in memory, typically in line or parallel with business processes. Its core capabilities are the ability to analyze massive quantities of data in real time, event by event.

“ While this may be the goal, in many businesses today the critical event remains elusive due to the volumes of data involved.”

Typically, this means looking for scenarios of events (patterns and combinations of events in succession) which are significant for the business problem in hand. The outputs of these systems are usually real time metrics; real time alerts; and the initiation of real time actions in custom or third party applications. The net effect, is that analysis processes are automated and do not rely on human interpretation and action. This real time, automated and continuous analysis provides dramatic returns and competitive advantage for those that have adopted it.

So how does it work?

If you consider where data is in your organization at any point in time, it resides in applications, data warehouses (having been extracted from applications), and is flowing through middleware. Since extracting data from the source applications slows down their performance, BI 2.0 gets data “in-flight” directly from middleware.

“ This real time, automated and continuous analysis provides dramatic returns and competitive advantage for those that have adopted it.”

By tapping into a flow of events, typically from industry standard middleware, BI 2.0 can analyze these streams of data to perform millions of calculations in real time that would be unthinkable in a traditional data warehouse environment. Middleware such as IBM MQ, MS MQ, Tibco, JMS, any EAI or ESB platform, SAP’s Netweaver or indeed Cisco’s AON blade in a network environment, can easily create a stream of events for analysis, similar to a third party listening in on a conference call.

This analysis is performed in memory – not written out to disk – and compares every individual event to what is normal for that unique account, customer, SKU, IP address etc. to identify those that represent problems or opportunities. It also enables a trend within a stream to be monitored, and in the most advanced systems, to project this trend forward automatically to understand the impact on the business in the near future.

Scalability

The ability to analyze large quantities of data in real time sounds very expensive, and indeed it would be in the traditional data warehouse world if you chose to follow this route. But there are major differences in the way that data is analyzed in Event Stream Processing technologies which fundamentally change the economics: each event is analyzed individually, not as a batch, and this, together with advanced in-memory computation techniques enables very high throughput rates of events to be analyzed on commodity hardware.

For example, if you were to calculate the sum of transactions by unique credit card over the last 30 days, in the old world, this would involve extracting data from a data warehouse from the previous 30 days for each credit card involved. Let’s assume that that there are 10m credit cards involved, and that each card makes one transaction per day. In the old world your query would pull back 30 x 10,000,000 or a total of 300m rows of data. This sequential processing would then involve 10m sums of 30 data points. If the same calculation was needed every time a new transaction is made, the entire calculation will have to be repeated.

In the BI 2.0 world, this doesn’t happen. Using ‘delta math’, products like SeeWhy calculate the impact of the changes on the sum. So while 10m sums are being held in memory, when a transaction happens on one card, the individual sum for that card is calculated by adding the new value onto the existing total, and dropping off any values from the other end of the sum as appropriate. Moreover, advanced caching techniques mean that the values in between the end points don’t need to be stored, dramatically increasing efficiency, and reducing memory usage. So in fact you don’t need to store all 10m data points to do the calculation.

“ Knowing there is a problem sooner buys the organization more reaction time. You can act while the customer is still on the phone, or on the website.”

The end result of these different techniques for doing these calculations means that Event Stream Processing engines are very efficient, and can typically process data an order of magnitude faster than query based approaches. This speed means that calculations can be done automatically as part of a process, whereas before this could not be considered.

If you can calculate metrics at an individual level, really fast, and in real time, then why would you introduce any latency? Concepts such as ‘right time’ have only been introduced because of the difficulty traditional architectures have in doing real time calculations.

Once this restriction is removed, then real time is the obvious route: after all, knowing there is a problem sooner buys the organization more reaction time. You can act while the customer is still on the phone, or on the website. Or perhaps a critical re-order cut off point is met, meaning that new supplies will be delivered to the store the next day. Clearly real time is the obvious goal of most organizations; it’s just traditionally been very hard and very expensive.

BI 2.0 changes all that for good. There is no reason to rely on out of date information any more.

by Charles Nicholls from "In Search of Insight" for more information visit www.seewhy.com

Custom Search

Send mail to webmaster@infohatch.com with questions or comments about this web site. Copyright © 2004-2012 Infohatch by DMS, All Rights Reserved. Last modified: 01/29/12

Send mail to webmaster@infohatch.com with questions or comments about this web site.
Copyright © 2004-2012 Infohatch by DMS, All Rights Reserved.
Last modified: 01/29/12