Data for Artificial Intelligence and Machine Learning - Part 1




Data for Machine Learning

 Raw Data
  • Raw data is a term used to describe data that is collected and stored, but not yet been processed.
  • Data is often defined as facts or numbers, but it can also be non-numeric and nonfactual.
  • It can be quantitative or qualitative data.
  • Raw Data can be collected from Sensors, User interactions with machine machine interfaces (e.g.,
  • Mobile devices, touch screens, keyboards,  websites etc.), Journal entries, event recordings, references information, medical diagnosis, recordings of human observations, experiments, communication, market behaviour, business transactions, social media and so on.)
  •  Raw data is sometimes called source data or atomic data.

Raw Data – Basic classifications

Fundamentally-2 classes – Primary and Secondary  - Both primary and secondary data may be quantitative and qualitative.

Primary data consists of a collection of original data collected by the researcher. It is often undertaken after the researcher has gained some insight into the problem by reviewing existing research work or by analysing previously collected primary data.


Secondary Data
Data that has previously been collected (primary data) by someone other than the current user. Secondary data is often used in social and economic analysis, especially when access to primary data is unavailable. Secondary data is the data that have been already collected and readily available from other sources.

Data Stores, Warehouse and Marts
Actually, only a small fraction of the data that are captured, processed and stored is used by decision makers. The data store, warehouse and marts provide facilities to integrate the data generated by an un-integrated environment.  A data warehouse organizes and stores all available data for further analytical process, information extraction and store its historical perspectives. 

Metadata
Metadata is data that describes the real data. Metadata describes the characteristics of data such as what are the identities of data, storage location details, where they originated from and how they can be accessed. In order to provide easy access, it is necessary to maintain a form of data directory with data information about data. Metadata are abstractions from data or high level data that provide concise description. It forms an important component of DW environment. 

Consider the following three different groups of data.
1.      23489, 33765, 27668.
2.      Auto magazine reports that vehicle sales has dipped by 25% in India during 2019 first half. 
3.      Sales figures of leading manufactures have dipped, Maruti 20%, Hyundai 30%, Mahindra 25%.

The first group of data conveys no information. The second group is more descriptive text and straightforward. The third one is concise and contains metadata.

Consider the case of book manufactures, a sample example of metadata is follows:
·         Author, title, ISBN number, publisher name
·         Headlines of stories
·         Definitions of data elements in a DW.
·         Road maps
·         etc


References: 
1. Wikipedia
2. Raw is an Oxymoron, L.Gitelman, MIT Press, 2013
3. Collecting Primary Data A time saving guide, Helen Kara, Policy Press, 2013



Comments

Post a Comment

Popular posts from this blog

Artificial Intelligence and Machine Learning Life Cycle

Image
The following is a series of four articles as PowerPoint slides in this blog site   History of Artificial Intelligence, Machine Learning and Deep Learning in slides Technologies Driving Artificial Intelligence Artificial Intelligence and Machine Learning Life Cycle Applications of Artificial Intelligence A to Z The slides are collections from my presentations at various skill development programmes.  These are meant for the benefit and information of AI enthusiasts, developers and researchers as well, to be aware of those great men and women who contributed to this field of human knowledge,  the key technologies that enable its progress, the general life cycle nature of AI projects and the wide variety of its present and future applications.  Comments from readers are welcome to improve the content.  
Read more

Modeling Threshold Logic Neural Networks: McCulloch-Pitts model and Rosenblatt’s Perceptrons

Image
Biological neurons exhibit a spiking output behavior in response to its inputs (refer Figures 10, 11 of " Essential biology and electrochemistry for Deep Learning " ). Inputs signals from the dendrites collectively raise the internal electric potential of the cell. Once this internal potential reaches or crosses the threshold value  the neuron fires. This is a nonlinear behavior During the formative years ANNs (1943-1958), McCulloch-Pitts introduced the idea of neural networks as computing machines. Frank Rosenblatt in 1962, and later Minsky and Papert in 1988 developed a large class of artificial neural networks called Perceptrons. McCulloch-Pitts model Warren Sturgis. McCulloch a neurophysiologist and Walter Harris Pitts a logician published a paper “A logical calculus of the ideas immanent in nervous activity”, 1943 which introduced the simplest artificial model of the biological neuron. McCulloch and Pitts modeled the nonlinear behavior with threshold logic function. ...
Read more

Regularization and Generalization in Deep Learning

Image
About Regularization and Generalization Regularization is one of the most important concepts in machine learning. In mathematics and statistics, finance, computer science, machine learning and inverse problems regularization is the process of adding information to solve an ill-posed problem. In the context of machine learning optimization problems, it applies a modification to the objective functions to reduce generalization error of the learning model even at the cost of increased training error. Generalization refers to the capability of a trained model to make the right predictions when faced with unknown input data during its operational life. In the rest of this article, we try to gain an intuitive understanding of the mathematical basis of regularization theory for inverse problems and its application to improve the generalization performance of learning algorithms. I have included some mathematical content to emphasize that, there is sufficiently strong mathematical founda...
Read more