We as humans find it easy to
understand something when it is represented in a logical manner. On the other
hand, computers and other machines need data to be organized in a structured
manner to understand it. Structured relational data is the data represented in
a structured way according to some specific rules that make it easy to
understand, analyze and interpret. On the other hand, unstructured data refers
to information without a predefined structure (obvious… isn’t it?). Computers don’t
care if the data is logical or not. All they need is for the data to be
organized in a particular manner. On the other hand, we humans can understand
unorganized data like text, images, videos etc. Now-a-days, there are
technologies with the capability to work on unstructured data. This has opened
up new avenues for mankind as data previously considered unusable for computers
can now be used. We can gleam new insights from it.
For structured data, languages
like SQL are used to retrieve data. As we can guess, SQL presumes that the data
is arranged in a structured manner. The statements used to retrieve data from
database using SQL have a specific format (known as syntax). Although there are
other languages/tools that are used to retrieve data from structured relational
tables using common language phrases/sentences like fetch me X, Y and Z from
table t1 where condition c1 happens, they internally rely on the data being
structured.
Talking about space requirements,
structured relational data requires very little space as compared to
unstructured data as structured data is designed to be used by machines. But,
there have been 2 major changes in technology today that tilt the balance in
favor of unstructured data: technologies to analyze unstructured data AND cheap
storage. Owing to organization within a structure, structured data is easy to
work with – for manual queries as well as automated queries (like search
engines). But trying to understand unstructured data requires more than just a
few queries. It requires one of the new functions of Web 3.0 – NLP (Natural
language processing) which is a whole new game altogether. Examples of
structure data include tables and spreadsheets. Examples of unstructured data
are sensory data, call records data, images etc.
For most of the organizations, technology
plays a supporting role. It is useful only if it can help the business gain
strategic advantage over their peers. The technology to manage/analyze
unstructured data came into place just a few years back. That is why, we now
see organizations starting to adopt such technologies. Just looking at the
volume of different types of data with organizations today:
Here, block based capacity refers to Structured data and file-based capacity refers to unstructured data
Technically speaking, a data
warehouse is a system used for reporting and data analysis. They are
centralized collection of integrated sources from variety of disparate sources.
Traditionally, data warehouses were designed to work only with structured data.
The sources of data as well as the methods used to work on them were based on
the assumption that the data is structured. But now, the situation is
different. Approximately 80% of business data is unstructured. And as expected,
most of this 80% does not reside in a standard relational database. Businesses
want to be able to rapidly analyze unstructured and structured data. Trying to
store huge volume of unstructured data inside the traditional warehouse is a
very tedious and time-consuming job.
If organizations want to deal with unstructured data, they either need to redesign their database from scratch, or they need to upgrade the existing databases and tools. They may have to start using/customizing big data tools and technologies like Hadoop and MapReduce. Although it is the matter of technology upgrade, the decision affects the organizations in critical ways. Hence, the decision has to involve business teams as well as technology teams. They have to incorporate integration techniques for all the processes from ETL to advanced analytics. They need to add metadata details for the unstructured data and that, is a very tedious process. Also care needs to be taken to avoid creating a data junkyard. New processes and systems for managing the documents has to be put in place. Techniques like the backward pointers might have to be used. Below is an example of how text analysis might be used
As fancy as it may sound, a data
warehouse is not an elixir that could solve all the problems. There are some
issues with analyzing data from a data warehouse
- Structured data is easy to manage in terms of ownership, security and privacy. But it is difficult to do so for unstructured data.
- Storage and reteival of unstructured data from a warehouse is very difficult and time consuming
- The amount of storage required for unstructured data is far more than what is required for structured data. An indirect implication of this is more time required to clean, filter and transform the unstructured data
Talking about
the future of data warehouse, I feel that the traditional data warehouse will
be highly integrated with the unstructured data warehouse. The primary purpose
of the data warehouse will be to handle unstructured data. There will be more
complex tools to work on unstructured data and they would also be integrated
with the data warehouse. One major drawback of data warehouses today is the
lack of real time analytics. This might not be true in future. With the advent
of real-time data processing softwares, it seems plausible that the data
warehouses in future would provide real time functionalities. Also, it is safe
to assume that the data warehouse might be stored on clouds instead of
traditional storage houses. Optimized warehouse is another thing that we can expect in future. Current warehouses are designed to work on static data. With unstructured data being stored in the database and the possibility of real-time analytics, we can expect future data warehouses to be highly optimized - for storage as well as for speed.
Overall, I feel
that the data warehouse industry is poised to grow a lot and will go through a
lot of changes in future. It will be a future that we all would like…
References:
No comments:
Post a Comment