A Dive in Big Data
Data is information in raw format. With increasing data size, it has become need for inspecting, cleaning, transforming, and modeling data with the goal of finding useful information, making conclusions, and supporting decision making. This process is known as Big Data data analysis.
Data mining is a particular data analysis technique where modeling and knowledge discovery for predictive rather than purely descriptive purposes is focused. Business intelligence covers data analysis that relies heavily on aggregation, focusing on business information. In statistical applications, some people divide business analytics into descriptive statistics, exploratory data analysis (EDA), and confirmatory data analysis (CDA). EDA focuses on discovering new features in the data and CDA focuses on confirming or falsifying existing hypotheses. Predictive analytics does forecasting or classification by focusing on statistical or structural models while in text analytics, statistical, linguistic and structural techniques are applied to extract and classify information from textual sources, a species of unstructured data. All are varieties of data analysis.
The Big Data wave has changed ways in which industries function. With Big Data has emerged the requirement to implement advanced analytics to it. Now experts can make more accurate and profitable decisions.
In this session of Big Data Analytics tutorial for beginners, we are going to see characteristics and need of data analysis.
Analysis versus Reporting
An analysis is an interactive process of a person tackling a problem, finding the data required to get an answer, analyzing that data, and interpreting the results in order to provide a recommendation for action.
A reporting environment or a business intelligence (BI) environment involves calling and execution of reports. The outputs are then printed in the desired form. Reporting refers to the process of organizing and summarizing data in an easily readable format to communicate important information. Reports help organizations in monitoring different areas of a performance and improving customer satisfaction. In other words, you can consider reporting as the process of converting raw data into useful information, while analysis transforms information into insights.
Let us understand difference between data analysis and data reporting in this Big Data Analytics Tutorial:
Reporting provides data. A report will show the user what had happened in the past, to avoid inferences and help to get a feel of the data while analysis provides answers for any question or issue.An analysis process takes any steps needed to get the answers to those questions.
Reporting just provides the data that is asked for while analysis provides the information or the answer that is actually needed.
Reporting is done in standardized manner while analysis can be customized. There are fixed standard formats for reporting while analysis is done as per the requirement and it is customizable as needed.
Reporting can be done using a tool and it generally does not involve any person while in analysis, person is required who is doing analysis and who will lead the process. He guides the complete analysis process.
Reporting is inflexible while analysis is flexible. Reporting provides no or limited context about what’s happening in the data and hence is inflexible while analysis emphasizes data points that are significant, unique, or special, and it explains why they are important to the business.
Any doubt yet in the Big Data Analytics tutorial for beginners? Please Comment.
Data Analytics Process
Now in Big Data Analytics Tutorial we are going to see the analytic process or how analyzing data can be done?
Data Analytics Process
Big Data Analytics Tutorial for beginners — Process
a. Business Understanding
The very first step consists of business understanding. Whenever any requirement occurs, firstly we need to determine business objective, assess the situation, determine data mining goals and then produce the project plan as per the requirement. Business objectives are defined in this phase.
b. Data Exploration
Second step consists of Data understanding. For further process, we need to gather initial data, describe and explore the data and verify data quality to ensure it contains the data we require. Data collected from the various sources is described in terms of its application and need for the project in this phase. This is also known as data exploration. This is necessary to verify the quality of data collected.
c. Data Preparation
Next come Data preparation. From the data collected in last step, we need to select data as per the need, clean it, construct it to get useful information and then integrate it all. Finally we need to format the data to get appropriate data. Data is selected, cleaned, and integrated in the format finalized for the analysis in this phase.
d. Data Modeling
Once data is gathered, we need to do data modeling. For this, we need to select modeling technique, generate test design, build model and assess the model built. Data model is build to analyze relationships between various selected objects in the data, test cases are built for assessing the model and model is tested and implemented on the data in this phase.
e. Data Evaluation
Next come data evaluation where we evaluate the results generated in last step, review the scope of error and determine next steps that need to be performed. Results of the test cases are evaluated and reviewed for the scope of error in this phase.
f. Deployment
Final step in analytic process is deployment. Here we need to plan the deployment and monitoring and maintenance, we need to produce final report and review the project. Results of the analysis are deployed in this phase. This is also known as reviewing of the project.
The complete above process is known as business analytics process.
Introduction to Data Mining
Data mining, also called as data or knowledge discovery, means analyzing data from different perspectives and summarizing it into useful information — information that can be used to take important decisions. And so we are discussing it in this Big Data Analytics tutorial. It is the technique of exploring, analyzing, and detecting patterns in large amounts of data. Goal of data mining is either data classification or data prediction. In classification, data is sorted into groups while in prediction, value of a continuous variable is predicted.
In today’s world, data mining is been used in several sectors like Retail, sales analytics, Financial, Communication, Marketing Organizations etc. For example, a marketer may want to find who did and did not respond to a promotion. In prediction, the idea is to predict the value of a continuous (ie non-discrete) variable; for example, a marketer may be interested in finding who will respond to a promotion.
Some examples of Data Mining are:
a. Classification of trees
These are Tree-shaped structures that represent sets of decisions.
b. Logistic regression
It predicts the probability of an outcome that can only have two values.
c. Neural networks
These are non-linear predictive models that resemble biological neural networks in structure and learn through training.
d. Clustering techniques like the K-nearest neighbors
This is the technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes it is called the k-nearest neighbor technique.
e. Anomaly detection
It is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset.
After this Big Data Analytics tutorial, you can read our detailed tutorial on Data Mining.
Characteristics of Big Data Analysis
We have already seen characteristics of Big Data like volume, velocity and variety. Let us now see in this Big Data Analytics Tutorial, characteristics of Big Data Analytics which make it different from traditional kind of analysis.
Characteristics of Big Data Analytics
Big Data Analytics Tutorial — Characteristics
Big Data analysis has the following characteristics:
a. Programmatic
There might be need to write program for data analysis by using code to manipulate it or do any kind of exploration because of the scale of the data.
b. Data-driven
A lot of data scientists depend on a hypothesis-driven approach to data analysis. For appropriate data analysis, one can also avail the data to foster analysis. This can be of significant advantage when there is a large amount of data. For example – machine learning approaches can be used in place of hypothetical analysis.
Get to know about the Top Data Science Skills for becoming a Data Scientist
c. Attributes usage
For proper and accurate analysis of data, it can use a lot of attributes. In the past, analysts dealt with hundreds of attributes or characteristics of the data source. With Big Data, there are now thousands of attributes and millions of observations.
d. Iterative
As whole data is broken into samples and samples are then analyzed, therefore data analytics can be iterative in nature. Better compute power enables iteration of the models until data analysts are satisfied. This has led to the development of new applications designed for addressing analysis requirements and time frames.
Applications of Data Analysis
Following are some of the popular applications of data analysis:
1. Fraud Detection & Risk Analytics
In banking, data analytics is heavily utilized for analyzing anomalous transaction and customer details. Banks also use data analytics to analyze loan defaulters and credit scores for their customers in order to minimize losses and prevent frauds.
2. Optimizing Transport Routes
Companies like Uber and Ola are heavily dependent on data analytics to optimize routes and fare for their customers. They use an analytical platform that analyzes the best route and calculates percentage rise and drop in taxi fares based on several parameters.
3. Providing Better Healthcare
With the help of data analytics, hospitals and healthcare centres are able to predict early onset of chronic diseases. They are able to predict diseases that might occur in the future and help the patients to take early action that would help them to reduce medical expenditure.
4. Managing Energy Expenditure
Public-sector energy companies are using data analytics to monitor the usage of energy by households and industries. Based on the usage patterns, they are optimizing energy supply in order to reduce costs and cut down on energy consumption.
5. Improving Search Results
Companies like Google are using data analytics to provide search results to users based on their preferences and search history. Furthermore, companies like Airbnb use search analytics to provide the best accommodation to its customers. Companies like Amazon are making use of the search analytics to provide personalised recommendations to its users.
6. Optimization of Logistics
Various companies are relying on Big Data Analytics to analyze supply chains and reduce latency in logistics. Amazon is making use of consumer analytics to analyze the customer requirements and direct them the products without creating any form of delay.
Summary
We discussed all the aspects of Data Analytics in this tutorial. Moreover, we looked at the difference between data analysis and data reporting with Data Analysis process, its types, characteristics and applications.
“This is yet just the surface”