I) Introduction - Big Data
- shift focus in IT from 't' to 'I', ie how you use the "information" will be the core aspect
- by analysing data with algorithms, we are finding patterns that no-one thought existed & these are challenging orthodoxies
- It is challenging our understanding of decision-making & comprehension of reality, ie from causality to simple correlations, ie not knowing why, only what happens, eg correlation between computer time and gross domestic product. The busier the road, the stronger the economic growth
- we can now rely on correlations without prejudgments & prejudice
How we value knowledge is changing - because we previously lacked enough, or right, data, we relied on intuition & experience and/or used random, representative sampling techniques. This has changed with "digital data", ie data & its analysis is not the limitation. No longer need to use sampling & then extrapolate to population
(NB information can be lost when we use sampling)
Allows "messiness", ie approximate is close enough, eg 2 + 2 = 3.9
Statistical analyses (including standardised factual questions, algorithms, etc) can be superior to personal judgments !!
You can't imagine what you don't measure.
More and more data, information, etc is available for organisations to know about their business; that knowledge can be harnessed and translated into improved decision-making and performance, eg online retailers can track what customers bought but also what else they looked at, how they navigated through the site, influence of promotions, reviews, testimonials, pay layouts, similarities across individuals, groups, etc.
Uses algorithms to predict activities and they improve their performance every time a customer responds. It provides an opportunity for competitive advantage and decision-making not dominated by intuition.
It will change our ideas about the value of experience, nature or expertise and the practice of management. There are 3 key elements (volume, velocity & variety)
i) Volume (more data crosses the internet every second than was stored in the entire internet 20 years ago. In 2012 around 2.5 exabytes of data are created daily and that number is doubling every 40 months (Andrew McAfee, 2012); it is estimated that Walmart collects 2.5 petabytes of data per hour from customer transactions.) (A petabyte is a quadrillion bytes; an Exabyte is 1,000 times a petabyte or 1 billion gigabytes)
ii) Velocity (real-time data/information, eg number of watching a TV show; at a shopping centre by phone locations, etc). Twitter updates were as accurate as official reports on tracking the spread of cholera in Haiti after the 2010 earthquake hit 2 weeks earlier
iii) Variety (basis for data is messages, updates, images, etc posted on social media (Facebook, Twitter, etc) plus mobile phones, GPS, etc. Recent reductions in computing costs like storage, memory, processing, bandwidth, etc is allowing for data-intensive approaches to be economical.
Research has shown that firms that are data driven perform better, ie 5% more productive and 6% more profitable.
Questions that need to be asked around the data, ie
- What does the data say?
- Where did the data come from?
- What kinds of analyses were conducted?
- How confident are we in the results?
Shift focus in IT from 't' to 'I', ie how you use the information will be the core aspect. Just as by using the telescope, we understood the universe better, and with the micro-scope, we understood germs better, Big Data is opening opportunities to other previously unthinkable possibilities
By analysing data with algorithms, we are finding patterns that no-one thought existed & these are challenging orthodoxies. It is challenging our understanding of decision-making & how we comprehend reality, ie from causality to simple correlations, ie not knowing why, only what. At the same time, we can now rely on correlations without prejudgments & prejudice
As mobile carriers realise that their future lies in data, they have offered unlimited national calling packages to remove revenue opportunities from apps that want to compete. Voice calls are becoming integrated with broader messaging categories like SMSs, chat messages, photos, MMSs, video & audio calls, recorded voice messages, etc
How we value knowledge is changing - because we previously lacked complete, or right data, we relied on intuition & experience. This has changed with "big data", ie data is not the limitation
Signals the end of supremacy of the subject matter specialist, eg media, universities, etc. & flags the increasing status of the statistician & data analyst, ie let the data speak, eg
- tabloid press - the data reveals what people want to read better than the instincts of seasoned journalists
- Amazon does not have book reviewers; instead it uses the data revealed from algorithmic findings of clients' activities & comments to drive sales
- TV programmers now get ratings in real time
- an on-line university course (coursera) studies data on the popularity of lecture, videos, etc to feed back into the course design
- a physicist developed algorithms to predict insurance claims & identify defective used cars
- an actuary predicted biological responses from chemical compounds
- how we value knowledge is changing - as we lacked enough, or right, data previously, we relied on intuition & experience. This has changed with "big data", ie data is not the limitation
- challenging our understanding of decision-making & comprehension of reality, ie from causality to simple correlations, ie rely on correlations without prejudgments & prejudice