Data Science with Spark

-

 DATA SCIENCE AND APACHE SPARK


Data Science has transformed the world. It has contributed towards the excessive growth of data and to develop intelligent systems. To analyze large amounts of data, various Data Science tools are available to Data Scientists. Among several available tools, Apache Spark has revolutionized the Data Science industry in a great manner.


Apache Spark

Spark is one of the open-source which is capable to process huge amount of data efficiently with very high speed. Due to its data streaming capability, Spark has left behind the other existing Big Data platforms. It also carry out machine learning operations and SQL workloads that allow us to access the datasets. Spark is developed on application levels through multiple languages like Python, Java, R, and Scala.

Components of Apache Spark for Data Science

Main components of Spark are – Spark Core, Spark SQL, Spark Streaming, Spark MLlib, Spark R and Spark GraphX.

SNo

Components

Description

1


Spark Core


  • This is the foundation block of Spark.

  • Contains an API that houses the Resilient Distributed Datasets or RDD.

  • Capable of carrying out memory management, storage system integration and fault recovery.

2


Spark SQL


  • To perform querying and processing of structured data.

  • Can also avail it for unstructured data.

  • Through this, HIVE, JSON, table can be accessed.

3


Spark Streaming


  • Ideal for many industrial applications and big data platform.

  • Easy manipulations of data stored in disks.

  • Use of micro-batching allows real-time streaming of data.

4


MLlib


  • Machine Learning is the ideal part of Data Science.

  • Spark contains a sub-project called MLlib to perform machine learning operations.

  • Operations like regression, classification and clustering can be performed.

5


GraphX


  • It’s a Spark library used for computing and manipulating graphs.

  • Clustering, classification, searching and path finding algorithms are used.

6


SparkR


  • Large datasets can be analyzed by using the R shell.

  • It makes use of the combination of the usability of R with its scalability.

Features of Spark for Data Science



Data Science with Spark

Apache Spark is well suited to handle unstructured information by providing a scalable distributed computing platform. In Data Science, it is used in the field of Text Analytics.

Techniques that are supported by Spark for text analytics can be described as–



Another important path of Data Science that is supported by Spark is Distributed Machine Learning. To 

support machine learning operations, Spark provides MLlib subproject.

Algorithms that are available within the MLlib project are –

  1. Classification

  2. Regression

  3. Collaborative Filtering

  4. Clustering

  5. Decomposition

  6. Optimization Techniques

Summary

Apache Spark is an ultra-fast, distributed framework for large-scale processing and machine learning. To support Data Science, it provides subprojects like MLlib, GraphX and also perform text analytics. It enhance the functionality of its machine learning library by adding features like Streaming and SQL services.

Thus, Spark is an idyllic platform for Data Science operations.






Author: 

Dr. Seema Maitrey 

Assistant Professor 

Department of Computer Science and Engineering 

KIET Group of Institutions, Ghaziabad




Resources:

  1. https://towardsdatascience.com/best-data-science-tools-for-data-scientists-75be64144a88

  2. https://www.researchgate.net/publication/339176824_Apache_Spark_A_Big_Data_Processing_Engine

  3. https://link.springer.com/article/10.1007/s41060-016-0027-9



Comments

  1. AnonymousOctober 26, 2021 at 11:22 AM

    Thank you for providing this article really appreciate the way you have put down the information if you want you can check
    data science course in bangalore
    data science course

    ReplyDelete

Post a Comment

Popular posts from this blog

QUANTUM STORAGE & MEMORY

-
Image
  Quantum computers stand to transform the world of computers. They have the capacity to be greatly more powerful than our current digital devices, with the power to solve important computational problems and simulate complex physical systems with unmatched efficiency. Like memories are an important component for computers, that’s like quantum memories are also essential components for quantum computers, a new generation of data processors that follow the laws of quantum mechanics and these laws help to overcome the limitations of classical computers. Quantum computers are expected to be much faster and more powerful than their traditional analogue as information is calculated in qubits (the fundamental unit of computation in quantum computing)  in quantum computers while the older units are bits which used in classical computers, can represent both 0 and 1 at the same time. Quantum memory is used as a key element in information processing, such as optical  quantum computing  and  qu
Read more

Biofuel generation with the help of Bioinformatics

-
Due to the continuously growing natality rate, increased availability of healthcare and food production has led to decreased mortality and further proliferated, population outburst on earth. This has caused strain on the energy sources and thereafter there has been an endless search for the most convenient, fast, and long-lasting method of renewable source of energy. According to the sources, from 2011 to 2021, renewable energy has grown from 20% to 28% of the global electricity supply. Fossil energy shrank from 68% to 62%, and nuclear from 12% to 10%. The share of hydropower decreased from 16% to 15% while power from sun and wind increased from 2% to 10%. Biomass and geothermal energy grew from 2% to 3% . Among many energy sources, Biomass can be one of the most trusted renewable energy sources, due to its abundance of availability present in nature. Biomass refers to organic material from plants or animals. This includes wood, sewage, and ethanol (which comes from corn or other plant
Read more

PHISHING ATTACKS IN INDIA

-
What do mean by phishing ?  Phishing is a type of online fraud when con artists pretend to be reputable companies or organizations in order to deceive victims into giving up personal information like passwords or credit card details. Phishing efforts can take the shape of emails, websites, or even text messages, and can appear authentic at first glance. A phishing attempt can be identified in a number of ways, including misspellings, odd syntax, and sudden requests for personal information. Do not respond to the message if you believe you are being targeted by a phisher; instead, report the incident to the relevant authorities. Phishing in India !!!! India is a vast and developing country. Digitalization is being done at a speedy rate. This brings a lot of ease for the general public but also creates a pool of opportunities for scammers, hackers, and con men. As per an article by Times of India Between April 1, 2020, and March 31, 2022, clients lost roughly 1500 crores in these scams.
Read more