Data Analysts vs Data Engineers vs Data Scientists




Data Analysts vs Data Engineers vs Data Scientists
What is data?
The word data refers to facts or numbers, collected to be examined and considered so as to help reasoning, calculations and decision-making.
Or

Data is information in an electronic form that can be transmitted, stored and used by a computer.

In modern computers data represented in binary form. “Binary data" in computers are actually sequences of bytes (combinations of 8 bits).



Figure - 1
Data Analyst
Data Analysis is the process of the extraction of information from a given pool of data.  

A data analyst is a person who engages in this form of analysis. 

A data analyst extracts the information through several methodologies like data cleaning, data conversion, and data modelling.  Various organizations analyse trends in the market, requirements of their clients and overview client performances with data analysis to take data-driven decisions.

Data analysts are responsible for taking actionable decisions that affect their organizations. They work with the management team to understand business requirements. The two most important techniques used in data analytics are descriptive (summary) statistics and inferential statistics.

A data analyst uses statistical modelling techniques that summarize the data through descriptive analysis and must be well versed with several visualization techniques and tools. As a result they must have good presentation skills to communicate the results with the team members and help them to reach proper solutions. Both data analysts and data scientists must be proficient in data visualization. Knowledge of machine learning is not important for data analysts but is essential for data scientists.

Figure – 2
Data Analyst – Skillset
In order to become a data analyst, you must possess the following skills –

  • Should have a strong suite of analytical and scripting skills. 
  • Should be strong in probability and statistics. 
  • Should be well versed with data visualization tools, and the methods of retrieving and manipulation data. 
  • Must have good knowledge of Excel, Tableau, Oracle, and SQL. 
  • Programming knowledge in either R or Python along with the use of popular packages.     
  • Must possess problem-solving attitude. 
  • Proficient in the communication of results to the team.

Data Engineer
Data Engineering  involves the development, construction of platforms and architectures for data processing. A Data Engineer is a person who specializes in preparing data for analytical usage. 

Data Engineers must necessarily possess knowledge of application development and working of the APIs. (An application programming interface is an interface or communication protocol between different parts of a computer program intended to simplify the development, implementation and maintenance of software. An API may be for a web-based system, operating system, database system, computer hardware, or software library. )

Data Engineer is responsible for creating, testing and maintaining the infrastructure required for storing and accessing past data quickly. Data Engineers have to deal with Big Data where they engage in numerous operations like data cleaning, management, transformation, eliminate data duplication etc.

In other words, a data engineer develops the foundation for various data operations. Therefore, they need expertise in both SQL and NoSQL databases. SQL databases are relational, table-based and vertically scalable databases. NoSQL databases non-relational, horizontally scalable and more cost-efficient databases. They do not require pre-defined schema and works with unstructured data. They are either document-based, key-value pairs, graph databases or wide-column stores. 


A Data Engineer is responsible for designing the format for data scientists and analysts to work on. For example, they may develop a cloud infrastructure to facilitate real-time analysis of data.


Figure – 3

Therefore, building an interface API is one of the job responsibilities of a data engineer. Data Engineers as well Data Scientists must work with both structured and unstructured data.

Data engineers need conceptual knowledge of machine learning methods, but not necessary require in depth knowledge. They are required to have the knowledge of core computing concepts like programming and algorithms to build robust data systems.

Therefore role of a data engineer also follows closely to that of a software engineer. Furthermore, a data engineer has a good knowledge of engineering and testing tools and must ensure data accuracy, flexibility and quality.

It is up to a data engineer to handle the entire pipelined architecture to handle log errors, agile testing, building fault-tolerant pipelines, administering databases and ensuring a stable pipeline, develop data processes for data modelling, mining, and data production.
A data engineer is generally not responsible for decision making.
Data  Engineer – Skillset
  • Knowledge of programming tools like Python and Java. 
  • Solid Understanding of Operating Systems. 
  • Ability to develop scalable ETL packages. 
  • Should be well versed in SQL as well as NoSQL technologies like Cassandra and MongoDB. 
  • Must possess knowledge of data warehouse and big data technologies like Hadoop, Hive, Pig, MapReduce techniques, Spark, Kubernetes, Java and Yarn.
  • Should possess creative and out of the box thinking.


Data Scientist
Data Science is currently the most trending job in the technology sector. While Data Science is still in its infantile stage, it has grown to occupy almost all the sectors of human businesses and related activities.
Data science is a multidisciplinary field dealing with data from diverse sources and includes all types of data operations like data extraction, data processing, data analysis to gain necessary insights and data prediction. These insights are used to make careful data-driven decisions.


Figure – 4
Moreover it is a quantitative field which includes applications with mathematics, probability, linear algebra, statistics, data mining and computer programming.
Data is available from anywhere and everywhere as a result, demand for data scientists who possess knowledge of learning tools and programming skills is on the increase. Organizations around the world are employ for skilled data personnel to improve performance and optimize their predictions and decisions. AI, ML and DL algorithms based operational models learns to extract information and hidden relations in large amount of data. These models are used to make predictions and also to discover patterns in the data. However, due to a relatively long learning curve, there is a shortage in supply for data scientists. This has resulted in a massive income bubble that provides the data scientists with lucrative salaries.
A data scientist is responsible for unearthing future insights from existing data and to help authorities take appropriate decisions. Hence data scientists participate in active decision-making process that affects the progress and growth of the organization.

Data Scientist – Skillset

For becoming a data scientist, you must have the following key skills –
·         Should be proficient with Math, Probability and Statistics.
·         Should be able to handle structured & unstructured information.
·         In-depth knowledge of tools like Python, R and SAS.
·         Well versed in various machine learning algorithms.
·         Have knowledge of SQL, NoSQL and Hadoop based analytics .
·         Must be familiar with Big Data tools.
·         Rock solid understanding of AI, ML and DL methods

Data Analyst Vs Data Engineer Vs Data Scientist – Salary Differences
·         On average, a Data Analyst earns an annual salary of $59,000
·         A Data Engineer earns $90,800 per annum
·         And a Data Scientist, on average, makes $91,400 in a year

Large corporations like Facebook, IBM etc quote upto $130,000 per year for data scientist.

Simple Tips of advises to AI/ML and Data Science students

1.       Learn any two languages preferably Python and Java.
2.       Develop expertise in a domain of interest.
3.       Improve communication and team working skills
4.       Evaluate media hype critically.
5.       Get connected to Linkedin/Facebook
6.       Give sufficient weightage to intuition as well as logic during problem solving
7.       Keep updated with developments in hardware as well as software technologies.

Figure credits:
Figure – 1: How data analysis is disrupting the legal industry
Figure – 2: https://www.quora.com/What-is- data-analytics
Figure – 3: Data engineering with databrick what, why, how
Figure – 4: https://datascience.aero/artificial-intelligence-vs-data-science/




Comments

Post a Comment

Popular posts from this blog

Artificial Intelligence and Machine Learning Life Cycle

Image
The following is a series of four articles as PowerPoint slides in this blog site   History of Artificial Intelligence, Machine Learning and Deep Learning in slides Technologies Driving Artificial Intelligence Artificial Intelligence and Machine Learning Life Cycle Applications of Artificial Intelligence A to Z The slides are collections from my presentations at various skill development programmes.  These are meant for the benefit and information of AI enthusiasts, developers and researchers as well, to be aware of those great men and women who contributed to this field of human knowledge,  the key technologies that enable its progress, the general life cycle nature of AI projects and the wide variety of its present and future applications.  Comments from readers are welcome to improve the content.  
Read more

Modeling Threshold Logic Neural Networks: McCulloch-Pitts model and Rosenblatt’s Perceptrons

Image
Biological neurons exhibit a spiking output behavior in response to its inputs (refer Figures 10, 11 of " Essential biology and electrochemistry for Deep Learning " ). Inputs signals from the dendrites collectively raise the internal electric potential of the cell. Once this internal potential reaches or crosses the threshold value  the neuron fires. This is a nonlinear behavior During the formative years ANNs (1943-1958), McCulloch-Pitts introduced the idea of neural networks as computing machines. Frank Rosenblatt in 1962, and later Minsky and Papert in 1988 developed a large class of artificial neural networks called Perceptrons. McCulloch-Pitts model Warren Sturgis. McCulloch a neurophysiologist and Walter Harris Pitts a logician published a paper “A logical calculus of the ideas immanent in nervous activity”, 1943 which introduced the simplest artificial model of the biological neuron. McCulloch and Pitts modeled the nonlinear behavior with threshold logic function. ...
Read more

Regularization and Generalization in Deep Learning

Image
About Regularization and Generalization Regularization is one of the most important concepts in machine learning. In mathematics and statistics, finance, computer science, machine learning and inverse problems regularization is the process of adding information to solve an ill-posed problem. In the context of machine learning optimization problems, it applies a modification to the objective functions to reduce generalization error of the learning model even at the cost of increased training error. Generalization refers to the capability of a trained model to make the right predictions when faced with unknown input data during its operational life. In the rest of this article, we try to gain an intuitive understanding of the mathematical basis of regularization theory for inverse problems and its application to improve the generalization performance of learning algorithms. I have included some mathematical content to emphasize that, there is sufficiently strong mathematical founda...
Read more