Yesterday, LucidEra, a pioneering vendor in the SaaS-based BI space, announced that it will be winding down its operations by the end of the month. In light of that announcement, it’s interesting to consider what it will really take for the on-demand, or SaaS-based, BI market to take off.

Conceptually, the idea of SaaS-based BI is a good one. It takes advantage cost-effective, low maintenance storage and compute capacity available in the cloud. It enables organizations to begin developing BI solutions without significant up-front investment. It promises superior ease-of-use, and it eliminates reliance on IT for solution delivery.

Other enterprise applications (Salesforce.com is the go-to example) have made it successfully to the cloud. What is holding BI back? In other words, what will it take for SaaS-based BI to be ready for prime time?

The way I see it, SaaS-based BI faces two major hurdles:

#1… SaaS-Based BI Hasn’t Solved the Data Integration Problem. The biggest challenge that users of SaaS-based BI face is how to get their data into the cloud. Most SaaS-based BI vendors support a simple flat file upload. Others offer primitive batch upload from a source RDBMS. These are fine options if all your data is in one nice, neat data set and that data set rarely changes or gets updated. But that’s rarely the case in the real world. Today’s SaaS-based solutions make unrealistic assumptions about the quality of the data that gets uploaded. There is no capability to do even simple data transforms or logical data mapping to establish data relationships. It is simply required that all fields across disparate sources to share the same name – a completely unrealistic assumption in the real world. There is also no recognition of the need to map data from multiple sources into one canonical data event for consolidated reporting and analysis purposes.

Because data loading is either done manually or, at best, in a scheduled batch, users find they are moving large amounts of data into the cloud unnecessarily and redundantly. Load times for large data sets are unavoidably slow. And there is no way to support any kind of low latency or real-time BI.

Suprisingly, these solutions have also failed to make it easy to integrate data in the cloud with data from on-premise apps. What should be an area of opportunity for on-demand BI solutions is in actuality a weakness.

And of course there are naturally issues around data security that are intrinsic to any cloud-based implementation. Are you really comfortable moving essential corporate data into the cloud? Even if you are already using other on-demand applications, there will likely always some corporate data you don’t want in the cloud under any circumstances. How do you govern and control this? Again, the simplistic tools offered to date rely on the user to solve the problems of redaction and other data reduction techniques to ensure only the data which is appropriate to cross the firewall is actually allowed in the cloud.

These are difficult and fundamental problems to solve. Solving them requires overcoming fundamental architectural hurdles. And even if that can be achieved, there is still a significant amount of design and development work that needs to happen from a user interface perspective to support the complex data transformation and integration challenges that are inherent to any serious BI solution.

#2… The End User Experience is Lacking. Right now, SaaS-based BI solutions deliver only very primitive data visualization and analysis features. Most fall far short of delivering highly interactive dashboards, with alerts, multi-path drill-downs, real-time data refresh, maps, synchronized component behavior, and other rich features. Eventually, they will bridge this gap. This functionality, is of course, all browser-based, so delivery is not an issue. Development is the barrier. Most traditional BI tools require a heavy client to build these rich dashboards – something that won’t work for a SaaS-based solution. But with enough time and effort, it is possible to design even the richest, most interactive dashboards in a browser. Altosoft, in fact, offers 100% browser-based dashboard and report development.

Customers seem to implicitly recognize these limitations. It’s true that many (typically smaller) organizations are playing with BI in the cloud. But adoption for serious BI projects, especially at major organizations, is negligible, or nonexistent. To overcome this the market will need to respond with a fresh approach to linking disparate systems and data sources behind the firewall with cloud-based deployment architectures making sure that the cloud-based tools (e.g. the visualization and analysis tools) meet or exceed those of their on-premise competitors.

One day, inevitably, the benefits of SaaS-based business intelligence will be realized. And the value will be real. But that day is not yet here. And it won’t get here by setting the bar so low that you can trip and still make it over. The tipping point will be when vendors offer solutions that raise the bar beyond the current on-premise offerings – not use the challenges of the cloud to excuse a lack of innovation.

It always surprises me how many buyers choose not to demand a POC from their vendor. I speculate there are a couple reasons for this:

1. Buyers think that POCs consume too much time and effort. They consume internal resources that just aren’t readily available; and
2. Buyers have conducted POCs in the past and have found that real product implementations have little relation to a vendor’s POC performance. So they dismiss the value of the POC.

To a certain extent, prospective buyers are right to think this way – with every POC there is both cost and risk to the buyer. That makes it all too easy for buyers to either talk themselves out of running a POC entirely, get convinced by a lazy vendor that a POC isn’t necessary, or run a halfhearted, token POC that gives the prospective vendor an easy layup win, but creates no value for the buyer.

Done right, a POC is invaluable. Rather than a source of cost and risk, it should be the ultimate cost saver and risk mitigator. Properly executed, it’s by far the best way find out whether a vendor’s sales claims are legitimate. Not only that, but there are other clear-cut benefits (quantifiable value) that you get from a POC:

1. A POC is a free, personalized product training session, underwritten by the vendor’s sales budget.
2. A successful POC should yield an impressive deliverable that can be used internally to gain support for an essential project.

If you are in the market for a business intelligence or data integration solution, it is a best practice to always run a proof of concept (POC) with your prospective vendor.

The trick is to constrain the POC effectively so your internal cost is limited, and to keep a sharp eye out for vendor sleight of hand and obfuscation — what my Irish grandmother would have called “shenanigans”.

AVOID VENDOR SHENANIGANS

How can you avoid shenanigans and get the most out of your POC? Here are a few helpful suggestions:

1. Insist that size doesn’t matter. If your vendor isn’t willing to do a POC because you, or your project, is deemed “too small” of an opportunity, it should tell you a couple things. First, it reveals how that vendor is likely to treat you in the future – as a low priority. Be wary. Second, it should make you suspicious that their solution isn’t as easy to use as they claim – because if it were really that easy, the vendor would have minimal downside (cost) and should be willing to do the POC.
2. Make the POC as real as possible. This is essential! If possible, test the vendor against your real data sources. Do not make it easy for the vendor by moving all your data in advance to a nice staging database with easy access! If it isn’t realistic to work directly against the data sources in the POC, do your best to come up with an environment that at least replicates the complexity of the real one.
   • Make sure you have more than one data source ready to go. Your project will probably will require analyzing data from more than one source. So should your POC. Most BI vendors can handle a single data source easily. Bringing together data from multiple sources gives them trouble. Test them.
3. Ask to see the queries. Make sure you get a good look at the queries they are running against your data.
   • How complex are the queries? How efficient? Are the queries optimized to minimize the burden on your source systems? Or will they need to rely on those systems to perform joins, aggregations, and calculations?
   • Does their architecture effectively isolate the source systems from the burden of identical, repetitive queries? Or will your production systems get hit with a new query every time a user refreshes a dashboard?
   • How much control do you have over when and how often queries run?
4. Take control. Make sure you drive the product installation, configuration, and solution design. Force the vendor to give you control of the mouse. Remember, it’s a learning experience for you. The best way to maximize it is to get hands on. Before the POC starts, you may want to work remotely with the vendor to do a clean product installation on your test machine to get things ready. There’s nothing wrong with that so long as you know what is involved in the install. As for the configuration and design – make sure you take ownership. That’s the only way to be sure you don’t fall victim to vendor tricks and clever shortcuts.
5. Ask questions. Lots of questions. Your best opportunity to get a straight answer is when you have control of the mouse.
6. Be prepared. Know the key metrics you want to measure in advance. Have a good understanding of how the metrics and data you want to see in dashboards and reports relate to your chosen data sources. But whatever you do, do not deliver a list of KPIs or a dashboard or report mockup to your vendor in advance. Make them work from a clean install! Don’t let them do any prep work. If you do, you open the door to vendor shenanigans.
7. Pay attention to the deployment process. Make sure you understand what it takes to move from product configuration into production. This will tell you a lot about the product’s architecture and ease-of-use.
8. Force the vendor to make changes. When you’re finally looking at pretty, interactive dashboards or reports, ask the vendor to start making changes. Because change is inevitable, you need to know what is involved.
   • Tell the vendor to add a new data source and join that data with existing data sources.
   • Have them add new dimensions.
   • Ask them to create new metrics, and change the way that current metrics are calculated.
   • Request that they change the manner in which data is visualized or presented to end users.
9. Operate on a timeline. At Altosoft, we typically do a one day POC. Sometimes two. Other vendors probably require more time. Ask your vendor how much time they need, negotiate this with them up front, and hold them to completing the POC in the requisite time frame!
10. Be prepared. A well designed POC should challenge your vendor. But you also need to hold up your part of the bargain. Be prepared for the POC. Have your test environment ready to go when your vendor arrives. If the vendor has provided a specification, make sure it is configured accordingly. And make sure that you are ready with the correct authentication to systems and data sources involved in the POC.

Now, here is what you absolutely should not do:

1. Don’t get suckered. The vendor may suggest that you export your data and send it to them before the POC so they can prepare. Don’t do it! Many vendors will take your data back to their lab, massage it, optimize it, have a team of experts build their solution in advance, and then script out the POC presentation like it’s a Broadway production. Classic vendor shenanigans!! Don’t allow it. Make the vendor work from real data, in your real systems, with no advance preview. Better yet, grab the mouse from the vendor and have them talk you through every step. No possibility of shenanigans there!

At Altosoft, we do a lot of POCs. After all, there’s not much downside to us. It’s easy to offer customers a free POC when you have confidence in your product, and you can use the product to create a great deal of value in a very short time. That’s what Altosoft is all about.

We typically offer a standard one-day POC session. The POC typically includes product install, connection to multiple data sources, data mapping, metrics definition, and development of a few dashboards and reports. We let the customer drive the process. It’s tight, well constrained, and people find it very productive.

THE VIRTUAL POC

But recently we’ve created a new POC variant variant that is faster and, for some customer prospects, even more exciting. We call this a “virtual POC”. Basically, it’s a POC that we deliver remotely, using a web meeting format. Here’s how it works:

We start off by helping you through the install process. This takes place prior to the actual POC web meeting. Installation is generally fast and easy.

Once you’re installed the product, we kick off the web meeting. Basically, over the next two hours, we talk you through the process of connecting to data sources, mapping data, defining metrics, setting up user permissions, deploying the solution, designing dashboards and reports, and finally visualizing and analyzing your data. It’s a fast, completely repeatable process. And since you’re driving, you get a great sense of how the product works. Of course, if you prefer, you can also hand control over to us for portions of the POC.

At the end of the process, we leave you with a working system. Not only that, but you have the knowledge and skills to extend it as you see fit. You can play with the system to your heart’s content.

The virtual POC is faster than the traditional POC, so the downside is that our prospects don’t get quite as much face-to-face training time. But it gives buyers a chance to get hands on with the product and experience how fast and easy delivering serious BI solutions can be with Altosoft.