At BG Bank by contrast we ended up more than 25 persons on the project and brought out only a draft datawarehouse with no end-user tools in a comparable period of time. Data volumes were somewhat larger but still.
This really got me thinking what a difference our choice of tools and methodology makes. In this comparison I would say at least a factor of 10. If we want this world to grow richer and better we must constantly find ways to do things more efficiently. If the same goal can be achieved with less effort then we are committed to aim for it in order to free up our ressources for better purposes - time for our children for example.
Since back then it has been my primary commitment to develop technologies and methods aimed at making implementation of business intelligence systems that:
- require minimum technical skills to work with
- are as fast and effortless as possible to implement and maintain
- provide the primary benefits of modern BI systems
Definition of LEAN BIThe main obstacles to efficiency and simplicity in business intelligence systems are:
- Datawarehouse, cubes, in-memory storage and other types of systems where data is copied from the operational system(s) to a separate data structure. Some vendors have other names for their offerings. We shall refer to all such structures under the common term Redundant Data Storage (RDS)
- The ETL processes required for building a RDS
- Complexity of modeling, ETL and end-user reporting tools
- Complexity of the source (ERP) systems
- Long development cycles
- Organizational and motivational issues
Let's have a look at these challenges from the perspective of a smaller company.
What I have found is that most small organizations have:
- All their primary data in one system, typically an ERP system
- Limited volumes of data
- Near-standard ERP systems
- Knowledge about the inner workings of their ERP system and SQL databases either in-house or through one of their primary service providers
Thus, in smaller organizations we don’t have the below reasons for creating an RDS:
- Integration of data from disparate sources
- Handling of large data volumes
We still have the problems with handling operational data schemas however, so in order to be able to eliminate the RDS we need new technologies that allow us to handle the operational data schemes inherent to operational systems.
The LEAN BI concept can be defined as these main areas:
- Use as few infrastructure components as possible
- Consolidation of end-users tools into as few products as possible
- Make development cycles as short as possible. Particularly the first cycle.
- Use of standard technologies
Sometimes the above requirements conflict because we are not living in an ideal world. Thus, striking the right balance between the above requirements is one of the most critical planning tasks in BI system development.
In a minimal and thus optimal LEAN scheme the BI system must:
- Have no middleware. I.e. it must handle operational data directly from the source systems
- Be based on one end-user tool
- Be implemented in a first prototype in one day
Is this possible?
Absolutely. Read on.
The case for LEAN BIBusiness Intelligence is an extremely useful (read: profitable) discipline for most types of organizations but the costs and complexity of building BI systems is often preventive for smaller organizations.
The main obstacles are:
- Cost of implementation
- Skill requirements, both technical and business skills
- Infrastructure complexity
- Cost of tools
LEAN BI provides a flexible framework that allows a BI system to be built to match your organization’s exact needs – not more, not less. If your source systems are simple then so should the BI system be. If you have a well-known source system then you should not have to re-invent the wheel for building a BI system. If your data volumes do not necessitate data replication then you should not have to replicate data. You get the idea.
What is LEAN?
Next post: Comparison of BI architectures