.
Statistical Methods of Big Data Analysis
- To deal with big data, some statistical tools are employed
- The tools to carry out basic data analysis includes
- Correlation Analysis
- Analysis of Variance and
- Hypothetical Testing
Correlation Analysis
- It is employed to identify linear relations among numerical variables
- It supports the exploratory data analysis
- But the correlation metric is not effected by outliers. It is named as spearman correlation
- This metric is robust and offers good determination of linear relations among numerical variables once the data is dispensed smoothly
- The underlying histogram anticipates distinction in the correlations of two metrics. Since the values are dispersed normally, the correlation of spearman is considered as good determination of the linear relation between the numerical variables
![Statistical Methods statistical methods img1](https://www.assignmenthelp.net/webimg/ahp/big-data-assignment-help/statistical-methods-img1.jpg)
![Statistical Methods statistical methods img2](https://www.assignmenthelp.net/webimg/ahp/big-data-assignment-help/statistical-methods-img2.jpg)
Chi square Test
- It offers to test 2 random variables are autonomous or not
- one distribution variable will not affect the another
- To assess the test in R tool , generate a table called contingency, then move the table to the R function in chisq.test
- To identify the link between cut variable and color variable from the diamond dataset. It is H0 and H1
- H0 represents the cut & diamond are autonomous variable
- H1 represents the cut & diamond are non autonomous variable
- The assumption is on relation between the 2 variables through its label but the test projects on rule by specifying the importance of the result
e.g.
T-test
- It is employed to verify whether there is any variance in a numerical variable and sharing among various groups of nominal variable
- To exhibit it, choose the grades of Fair and Ideal of the variable factor cut, and then evaluate the variable values between the 2 groups
Analysis of Variance
- It is shortly denoted as ANOVA coined by Ronald Fisher
- It is a statistical model employed to investigate the variance among diverged group by a contrasting each group mean and variance
- It offers an evaluation that the mean’s for different groups are same or different
- As a result, t test is generalized for greater than 2 groups
- It is good for comparative study of 3 or greater than 3 groups for finding the importance of statistical analysis
.