I long for the day that I can sit in the back of my car and read a book or take a nap. This leads me to look up articles on self-driving cars, and in the spirit of automation, why not automate my search.
For this program, you will need to have a simple knowledge of NodeJS. Nothing special, we’re just making a simple server and using Axios and Cheerio to get the data and create an API.
The first thing you need to do is open your terminal and create a nodejs project with …
npm init
then, install …
npm install express axios cheerio
This will update your package.json file and it should look like this …
Next, we need to import each of these into our index.js file.
const express = require(‘express’);
const axios = require(‘axios’);
const cheerio = require(‘cheerio);
const app = express();
Those are my first four lines of my program.
Next, you want to create a variable with all the periodicals that you will be scraping with cheerio.
I would recommend adding a lot more than this. I have 12 right now. Believe it or not, sometimes there isn’t a ton of new news on self-driving cars floating around the internet.
Create an Express Server
One of the best things about express is how easy it is to create a server and the routes you need.
You’re going to use the app variable that you created above create the server by typing.
You can make the port something other than 3001, in case you were wondering. Also, notice how the code lines are 130–133. That’s because I like to keep the server at the bottom.
At this point, I would encourage you to try running your server.
Go into the terminal and type … node index.js
If it works correctly your terminal should say …
your server is running on 3001
Create your Routes
For this section, you will need two routes.
Lines 96–98 are simply a welcome page, and the ‘/news’ route will be used for our results.
We’re using a GET request for both which is an http method that is used to retrieve information and display it to the page.
Disengage your server with ctrl+c. Then restart it with node index.js.
Then open up an empty browser window and type …
You should see the message below.
Retrieve Data from Sources
Now, we get to the most important part of this program, which is getting the articles.
So, what we are going to do is scrape links from popular newspapers, magazines, and blogs by searching the page for titles that have the word ‘self-driving’ in it. This will use Axios to make the get request of html data and Cheerio will look through that data to find the titles that we want.
If there is a match then that data will get pushed into an array of articles which will be displayed in the ‘/news’ route we created …
Here is the code to get the articles…
Starting from the top, declare a variable called articles that is set to an empty array.
Then iterate through the newspapers array of objects we created earlier, then use Axios to perform the get request from each address. Using a .then method, we will get the response back and then set the response.data as the variable ‘html.’
This is where we use the .load method supplied by Cheerio and pass in the html variable. This method parses the html data and makes it available for us to get what we want. For a more detailed explanation you can look at the source code.
By setting the cheerio.load(html) to a variable ‘$,’ we can now look through all the <a> tags (links) with the text ‘self-driving.’ Once we do this, we will set the information we get to variables of ‘title’ and ‘url’ and then push that information into the articles array. Remember, this array gets called on the ‘/news’ get request and will now show you the articles we received…
It looks like at this time, it’s a light day for self-driving news articles, but try it each day and you’ll get a plethora of articles over time. Still this way better, than searching all these websites manually.
One thing, I’ve noticed is that if you change the case of ‘self-driving’ to ‘Self-Driving’ or even ‘Self Driving’ or ‘self driving’ you can get different results.
Enjoy!