The Complete Truth of Life

demo

For DWD final, I made a web-based book named the complete truth of life.

visit the site

view source codes

Concept

The complete of truth does not exist. However, people have never stopped discussing and looking for answers. The book seeks to converge and curate wits from the internet that will eventually outlive us. This is a book that can only exist in the digital realm.

Process

Web-scraping:

I used Cheerio.js to scrape data from Quora, Reddit, and Yahoo Answers.

Screen Shot 2018-03-15 at 9.08.25 PM.png

Challenge:

Some websites such as Quora has “scroll down the page to load” function, which makes it difficult to do web-scraping. For example, say there are 700 answers under one question, but I was only able to scrape the first 7 answers because the full HTML doesn’t load until you scroll down the page. I was able to get around this problem by saving the html to local and scrape data from the local file. 

Save data to MongoDB

I used MongoDB and mogoJS to store and manage my data.

Screen Shot 2018-03-15 at 9.26.04 PM

Serve the website

The website was served using Express and ejs Template.

Screen Shot 2018-03-15 at 9.28.43 PM.png

Front-end

Front-end of the website was written with HTML/CSS/jquery and turn.js.

Screen Shot 2018-03-15 at 9.35.35 PMScreen Shot 2018-03-15 at 9.36.06 PM

Search Key Words

I created search function with query operators and regular expression.

Screen Shot 2018-03-15 at 9.38.36 PM.png

Next Step

  1. I wanted to make this live, meaning when people search a keyword, they get live-scraped “life advice” from the answer forum.
  2. As I worked on this project, I found the analogy between a real book and internet forum very interesting. Writers write books that would influence other people even after they die, and in the age of the internet, we all leave words and traces online that will eventually outlive ourselves. I would love to make series of books on different topics from internet data.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s