Can I make my own custom search engine from scratch?

-

The title is quite intriguing but interesting. As a student of computer science I had always thought how a search engine works, it knows about a newly commissioned web resource, whether a website is up or down, why a website takes time to show-up in search results? There are many questions, but the only answer is search engine. This is first article in series of search engine which will cover the basics, modules of search engine, development of the modules with code and results, integrating the modules with one framework.

We all use search engine daily like

ok Google when cab is going to arrive, when is my birthday?

Alexa will it rain today?

Search engines like Google and duckduckgo do same things, but ideologically different. So lets get down to basics of search engines and get our hands dirty with coding a basic search engine.


Basics

Search engines are a complex piece of code which does so many things like searching for the answer to the question user had asked, calculating which website comes in first 10 results. To explain we will now first understand different components of search engine and they are:

  • crawling
  • parsing
  • indexing
These three processes teams up to make a search engine and keep it up and running. Lets dive inside these components one by one.


Crawling



Crawling is a process which feeds the search engine with the required data. It first visits the page through internet, downloads the information and saves it to the database of the search engine. Some of the information a crawler seeks for are:

  • title of the website
  • url
  • keywords
  • meta information
  • domain information
  • javascript
  • CSS
  • headings
  • other links in the website
The software which does crawling is termed as spider or crawler. A crawler when plans to visit a website first checks for a file robots.txt on the server where the website is hosted. This file contain rules for the crawler. It informs crawler what to crawl and what not to crawl. After the crawler had read the rules then it crawls the index page of the website and jumps on to next website after it. These crawlers can’t search entire website in one go so they stop after scanning the first page(mostly).
Now I will make you understand the next process i.e., parsing.

Parsing

It is a process in which the webpage is scanned for the content inside it.
So how do we do it?
We do it by extracting the features from the webpage and store it with the handle i.e, web url. Some of the features of the websites are:

  • website domain name
  • website title
  • frequently used tokens(words)
  • headings on the page
  • links to other websites
  • links to other pages of the site
  • text marked as bold
  • thumbnails of images in website
These features are stored in a database. After storing, indexing is done based on indexing criteria. It is discussed in next section.

Indexing

It is a process of processing the data of the crawled websites and arranging them in an order, so that we can retrieve required information later. It is described in the figure 1 below.




In the Figure 1, text acquisition is done in parsing phase and data is stored in document store, text transformation is done which cleans text and then pushed again for index creation. Index creation is done after it.
I hope you have understood the basics of search engine.
In my next article I’ll explain each module in detail.



Gaurav Parashar (Faculty)

Computer Science Department

KIET Group of Institutions

Location: Monthly Wrap KIET Ghaziabad, Uttar Pradesh 201206, India

Comments

Post a Comment

Popular posts from this blog

QUANTUM STORAGE & MEMORY

-
Image
  Quantum computers stand to transform the world of computers. They have the capacity to be greatly more powerful than our current digital devices, with the power to solve important computational problems and simulate complex physical systems with unmatched efficiency. Like memories are an important component for computers, that’s like quantum memories are also essential components for quantum computers, a new generation of data processors that follow the laws of quantum mechanics and these laws help to overcome the limitations of classical computers. Quantum computers are expected to be much faster and more powerful than their traditional analogue as information is calculated in qubits (the fundamental unit of computation in quantum computing)  in quantum computers while the older units are bits which used in classical computers, can represent both 0 and 1 at the same time. Quantum memory is used as a key element in information processing, such as optical  quantum computing  and  qu
Read more

BLOCKCHAIN: HOW IT WORKS AND WHY SO POPULAR

-
Image
Merely, 12 years ago on 31 st October 2008, SATOSHI NAKAMOTO published a paper on the internet about the idea of bitcoin. The Paper’s first few lines almost depicted all. The lines – “A version of electronic cash that would allow payments to be sent directly from one party/person to another party without involving a financial institution.” The value of bitcoin in 2010 was ~ 0 INR and value in 2020 ~ 15 lakhs INR. Recently, it touched its high price point, and now again is the topic of discussion among the market and media. Why so much difference? Like the US dollar is controlled by the Central Bank of US and the Indian rupee is controlled by RBI. Normally, our financial systems are based on “TRUST”. Whereas Cryptocurrency is a digital asset over which central banks or financial institutions have no control or rules/regulation. The currency notes and coins have value just because they are guaranteed by the government and central bank. Every currency has a promissory note with the signa
Read more

QUANTUM COMPUTING: CAN FIGHT CLIMATE CHANGE

-
….IN AN INFINITE UNIVERSE ANYTHING THAT COULD BE IMAGINED MIGHT SOMEWHERE EXIST.                                                                   - Dean Koontz Quantum computing is an emerging domain that is yet to enter our lives. However, the researchers are working hard to bring it into existence. Quantum computing can make the impossible possible. There are some problems that even supercomputers can’t solve. Quantum computers will however have the potential to solve such problems. In the same way that bits are used in a classical computer, quantum computers use quantum bits or qubits which can store information in the quantum form. A qubit can be 0 or 1, or a superposition of both 0 and 1. Experts expect quantum computing to help us understand biology and evolution, cure cancer, and even take steps to reverse climate change. Imagine you have four bits. Together, those four bits can have one of 16 possible combinations, such as 1011.  The unique powers of quantum computation may g
Read more