LEAN Business Intelligence

The need for efficiency

I used to work for the large Danish companies FDB (now COOP) and BG Bank (now Danske Bank) implementing their first datawarehouse and BI systems. Both as a developer and as project manager. I was astonished by the difference in approach to implementing datawarehouse and business intelligence systems (and probably IT systems in general). At FDB we had a scarcity of ressources but accomplished a lot due to a "just do it" culture in our department plus a choice of technologies that made it fairly simple to setup load jobs, create reports, handle large volumes of data and so on. With less than a handful of developers - in one period I even worked the project alone - we succeeded to get a full datawarehouse/BI implementation rolling with data from all +1,000 supermarkets and a comprehensive end-user environment.

At BG Bank by contrast we ended up more than 25 persons on the project and brought out only a draft datawarehouse with no end-user tools in a comparable period of time. Data volumes were somewhat larger but still.

This really got me thinking what a difference our choice of tools and methodology makes. In this comparison I would say at least a factor of 10. If we want this world to grow richer and better we must constantly find ways to do things more efficiently. If the same goal can be achieved with less effort then we are committed to aim for it in order to free up our ressources for better purposes - time for our children for example.

Since back then it has been my primary commitment to develop technologies and methods aimed at making implementation of business intelligence systems that:

1. require minimum technical skills to work with
2. are as fast and effortless as possible to implement and maintain
3. provide the primary benefits of modern BI systems

In my work I have uncovered a range of technologies and methods that will help to achieve these goals. These technologies and methods will in turn allow more people and more companies to exploit the benefits of business intelligence than do traditional BI tools and methods.

Definition of LEAN BI

The main obstacles to efficiency and simplicity in business intelligence systems are:

• Datawarehouse, cubes, in-memory storage and other types of systems where data is copied from the operational system(s) to a separate data structure. Some vendors have other names for their offerings. We shall refer to all such structures under the common term Redundant Data Storage (RDS)
• The ETL processes required for building a RDS
• Complexity of modeling, ETL and end-user reporting tools
• Complexity of the source (ERP) systems
• Long development cycles
• Organizational and motivational issues

I shall restrict myself from dealing with organizational and motivational issues in this book, which has a technical perspective.

Let's have a look at these challenges from the perspective of a smaller company.
What I have found is that most small organizations have:

• All their primary data in one system, typically an ERP system
• Limited volumes of data
• Near-standard ERP systems
• Knowledge about the inner workings of their ERP system and SQL databases either in-house or through one of their primary service providers

Thus, in smaller organizations we don’t have the below reasons for creating an RDS:

• Integration of data from disparate sources
• Handling of large data volumes

It is therefore obvious to aim for elimination of the RDS and ETL processes, which account for typically 75% of the total costs.

We still have the problems with handling operational data schemas however, so in order to be able to eliminate the RDS we need new technologies that allow us to handle the operational data schemes inherent to operational systems.

The LEAN BI concept can be defined as these main areas:

1. Use as few infrastructure components as possible
2. Consolidation of end-users tools into as few products as possible
3. Make development cycles as short as possible. Particularly the first cycle.
4. Use of standard technologies

Sometimes the above requirements conflict because we are not living in an ideal world. Thus, striking the right balance between the above requirements is one of the most critical planning tasks in BI system development.

In a minimal and thus optimal LEAN scheme the BI system must:

1. Have no middleware. I.e. it must handle operational data directly from the source systems
2. Be based on one end-user tool
3. Be implemented in a first prototype in one day

Is this possible?

Absolutely. Read on.

The case for LEAN BI

Business Intelligence is an extremely useful (read: profitable) discipline for most types of organizations but the costs and complexity of building BI systems is often preventive for smaller organizations.

The main obstacles are:

1. Cost of implementation
2. Skill requirements, both technical and business skills
3. Infrastructure complexity
4. Cost of tools

The most important factor that contributes to these obstacles is the fact that common technologies and methodologies are aimed at large and complex organizations. Thus, when a vendor glorifies their products with superlatives like scalable then it usually means that its cost of implementation will be preventive in a small organization; it is meant for large organizations. I.e. scalable usually means upwards scalable, not downwards.

LEAN BI provides a flexible framework that allows a BI system to be built to match your organization’s exact needs – not more, not less. If your source systems are simple then so should the BI system be. If you have a well-known source system then you should not have to re-invent the wheel for building a BI system. If your data volumes do not necessitate data replication then you should not have to replicate data. You get the idea.

See also:
What is LEAN?

Next post: Comparison of BI architectures