References:
- The Google File System, Google Inc, 2003
- MapReduce, Google Inc, 2004
The goal of the first reference is the creation of a distributed file system and the second is distributed computing.
Data model
Data model has three types:
- Structured
- Semi-structured
- Unstructured
Structured Data
It has a fixed number of fields, no data entry has more or less fields than another entry, we can put null in fields where we don’t have data.
Elements can be easily be accessed through indexing and key relationships.
Scaling Data
In the past, structured data was scaled buying a computer with a very large hard disk and memory to store the data. Now we have better ways like databased clustering among many servers or using extended virtual storage RAID. Only some RAID modes allow the increase of capacity like RAID 0. Raid
Clustering
The data is available in many places but appears as one, we have only one DBMS.
Data Availability
- Have mirrored database that have the same data
- Use redundant storage, another RAID mode
So scaling and clustering was so complex, because I had to mirror every cluster unit in my network.
Semi-structured Data
Semi-structured data does not have a fixed structure, but they still have some kind of structure that they conform to, an example for that are XML and JSON.
The more the structure becomes variable the more we will look like unstructured data.
Joining is harder than RDBs because the we are not really confined to a scheme that has relations.
We have much more data analysis possibilities because we have intermediate stage where we process this data.
We can think of the intermediate as a layer that does some kind of analysis of our data or it “understands” the data, so we can do some kind of analysis on it.
To scale semi-structured we will use distributed file system.
Unstructured Data
This type of data does not adhere to any type of schema and there is no specific data format.
Here a more advanced data processing stage is needed because we don’t have a some kind of structure to go with, so we will need more advanced algorithms to “understand” this data.
Also there is no query model in unstructured data.
NOTE: So to apply visualization, I’ll have to transform the unstructured data to semi-structured and the semi-structured to structured data with relations.
