Big data are data sets of information that are so voluminous and complex that traditional data processing software is inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, searching the data, sharing it, transferring it, visualising it, querying it, updating it and most important of all protecting its privacy.

There are Two Main Types of Big Data

1. Unstructured data – that comes from unorganised information usually in the form of large amounts of text which is easily interpreted by traditional databases or data models. Metadata Facebook posts, Twitter tweets and other social media posts are good examples of unstructured data.
2. Multi-structured data – refers to a variety of data formats created by interactions between people and machines, such as web applications or social networks. A good example of this include combinations of text and visual images along with structured data like filling in online forms or purchase transaction information.

As technology continues to advance and enhance the customer experience multi-structured data will continue to evolve and expand.

The History of Big Data and Where it’s Going

Like with many technological advances, cool catch phrases and new marketing techniques play a big part of technological advances. So is the case with the birth of the term “big data”. Gathering and storing large amounts of information for analysis is not new at all, but it’s very different from what we’ve known and seen until now…

The current understanding of the term “Big Data” was first coined by Doug Laney who defined its three central dimensions:

• Volume – Organisations collect data from a variety of sources, including business transactions, social media and information from sensor or machine-to-machine data. In the past, storing it was a problem – but new technologies have eased the burden.
• Velocity – Data streams at a very high speed and must be dealt with efficiently. RFID tags, sensors and smart metering all drive the need to deal with torrents of data in near-real time.
•  Variety – Data comes in all types of formats – from structured and numeric data in traditional databases to unstructured text documents, emails, videos, audio, stock data and financial transactions.

There are two additional dimensions that are now also being included in the definition and understanding of big data, and they are:

•  Variability – In addition to the increasing velocities and varieties of data, data flows can be highly inconsistent with periodic peaks. For example, something that is trending in social media. Daily, seasonal and event-triggered peak data loads can be challenging to manage. Even more so with unstructured data.
• Data Complexity – Today’s data comes from multiple sources, which makes it difficult to link, match, cleanse and transform data across systems. However, it’s necessary to connect and correlate relationships, hierarchies and multiple data linkages or your data, unchecked it can quickly spiral out of control.

What Can We Do With All This Data?

The amount of data being created and collected on the web is inconceivable, and it keeps growing and expanding every second. This means that there are more key insights an understandings that can be gained from a business from the harvested data collected. 

Currently, only a small percentage of that data is analysed.

The importance of big data has nothing to do with the amount of data you have, but what you do with it. You can take data from any source and analyse it to find answers that enable you to save time and reduce costs on product development as well as optimised your product offering.

If you integrate big data with analytics, you can achieve improved business performance and increase revenue. This is achieved by finding the root causes of errors and defects almost on the fly.

Big Data and Data Analysts

The true value of big data heavily depends on the human data analysts tasked with understanding the data and then translating the gained insight into formulating the proper queries to direct big data projects.

Some big data offerings provide solutions designed for specialised niches and allow less technical users to make various predictions from everyday business data. Other offerings require higher level of understanding and minimise the need for customised software and hardware.

 Big Data in Day to Day Applications

Big data has increased the global demand of data scientists so much so, that major software and hardware vendors have spent more than $20 billion on software firms specialising in data management and analytics to date. The big data industry has grown to more than $194 billion (US) and was growing at approximately 10 percent per annum. This is approximately twice as fast as the growth in the software industry.

Advanced first world economies have also started using data-intensive technologies for a variety of purposes. For example, there are currently more than 5 billion mobile-phone subscriptions worldwide, and approximately 1.6 billion people accessing the internet daily. 

There are more than 1 billion people worldwide who are classed as in the middle class, which among other factors refer to their level of literacy. The literacy rate has a direct impact on information growth. Approximately one-third of the globally stored information is in the form of alphanumeric text and still image data, the most applicable and useful for big data applications.

Big Data and Manufacturing

The main impact of big data on manufacturing focuses mainly on improving the planning and product quality and supply chain. Big data provides transparency to the manufacturing industry, which is reflected in tracking the inconsistency of product components, performance and availability. Big data also enables predictive manufacturing for near-zero downtime.

 Government Use of Big Data

The use and adoption of big data within governmental processes enables increased efficiency in terms of cost, productivity, and innovation, but this does not come without a price as much of the information gathered has the potential of reducing a citizen’s privacy not to mention information theft.

Data analysis often requires the integration of multiple government departments to work in collaboration and create new and innovative processes to deliver the desired outcome which creates centralised data repositories and security flaws such as the government employee’s ability to view and potentially steal personal data. This factor of big data theft is as frightening as no one can really promise that it is secure.

 Additional Big Data Implementations

There are many ways that we can utilise big data to improve a broad range of daily function to improve our lives. I will briefly cover the main areas of interest in my opinion.

The Big Data of the Healthcare Systems

Big data analytics helps the healthcare industry improve the accuracy of personalised medication and risk analysis. It also helps standardise terminology across predictive analytics, medicine wastage and automated reporting for patient data. There is a huge amount of data generated in the health care industry and it will continue to increase.

This includes electronic health record data, imaging data, patient generated data, sensor data, and other forms of data that is difficult to process. Not to mention human chip implants that have begun to be implemented in Sweden for example, although this is also affected by cultural accepted norms, and religious limitations.

Certain societies will embrace this chip implant technology, while others will completely reject it.

 Big Data in the Media Industry

The media industry is moving away from the traditional forms of media such as newspapers, magazines, or TV shows and is tapping into consumers with technologies that reach targeted people at optimal times in optimal locations.

The media industries aim is to focus a message or content that is in line with the specific consumer’s mindset. Publishing companies are increasingly tailoring advertisements and content articles to consumers that have been personalised and filtered through various data-mining mechanisms.

The Next “Big Data” Thang – Internet of Things (lOT)

The next big thing that is going to happen in the big data arena is the jamming of micro sensors and internet connectivity into people, cars, clothing, shoes, toasters, surf boards, bicycles and just about anything that you can imagine.

In addition, the integration of smart devices will also morph into a integrated monster of big data. These internet connection items will be able to transmit data in real time to an array of databases generating massive amounts of big data, which will provide an incredible amount of consumer data.

“If we had computers that knew everything there was to know about things—using data they gathered without any help from us—we would be able to track and count everything, and greatly reduce waste, loss and cost. We would know when things needed replacing, repairing or recalling, and whether they were fresh or past their best.” Kevin Ashton

Big Data Analytics

Artificial Intelligence (AI), mobile, social and Internet of Things (IoT) are increasing data complexity and providing new forms and sources of data. Big data analytics is used together with advanced analytic methods on very large, diverse data sets that include structured, semi-structured and unstructured data, from different sources, and in different sizes from terabytes to zettabytes.

Analysing these new forms of big data will enable analysts, researchers, and business users to make more accurate and faster decisions using data that was previously inaccessible to them. By leveraging advanced analytics techniques such as text analytics, machine learning, predictive analytics, data mining, statistics and natural language processing businesses will accurately analyse previously untapped data sources gaining new insights and understandings, which will result in improved and more accurate decisions.

 The Big Brother of Big Data

Who would have thought when reading George Orwell’s classic 1984, that big brother wouldn’t be an external government entity forcing itself into every aspect of your life, on pain of death! Rather, a consumer stampede towards full compliance and total voluntarily integration into the web. Rushing to purchase interactive GPS cell phones, geo tagging themselves at every purchase point, uploading photos, hysterically entering personal data into online social network groups and exposing every aspect of their private life to the public.

Yes, 1984 has arrived and we the consumers can’t get enough. Now throw into the blender the power of AI, crypto currency and big data analytics and we begin to see that our freedom to live, consumer and move around the globe has completely disappeared. We have become completely transparent…. 

It is as frightening as it is exciting at the same time.