Creating my news application.
News News News
Newsfoldr is a news application I created using the React framework, it was created as part of my list of new year resolutions, and my frustrations with local news applications cramming up their site with a lot of ads, to squeeze out the possible amount of ad revenue. Surely there had to be a better way to keep people up to date and still make some change to buy lunch. With these in mind, I set out to build newsfoldr, clearly not because I think it would be the next big news application but rather to challenge myself and also be able to put React and React Native on my resume.
Creation Process
I started by coming up with a list of features, I would want my ideal news application to have, some of this included radio so I could listen to my favourite radio channel (YFM ~ they play all the new hip-pop beats here), and all relevant news, which was categorised in an easy to understand form these features are not in any way original since there are literally thousands of news websites that follow the same pattern. So how would newsfoldr be different, I thought of having newsfoldr update you passed on your news preferences, via various social media channels. This was largely influenced by a documentary I had watched in the previous weeks that stated how most people are spending a minimum of 1 hour every day on Facebook apps (Facebook, WhatsApp and Instagram). I thought if people already spend their time there, then it would be super difficult to redirect their attention to other platforms, hence why not meet the people where they are?
I resolved to create a solution where the users would have to interact with a messaging bot either through WhatsApp or telegram, there reason was I generally think it would be time-consuming to create bots for every platform because I couldn’t find much documentation on bot creation for the other platforms and creating a bot for these platforms will be an exercise in futility since people might not even decide to use it.
Application Creation Process
I had already decided to use React to create this application, so I listed the expected functionalities of this project, which were mainly
- Get news data from some source.
- Store the news in some database.
- Create a bridge through which that data will flow to a given interface.
- Create a set of interfaces(bot, mobile app, web app) through which data will be displayed.
The above list shows the set of major problems I had to solve in order to finish this project, I then broke down each problem and placed them on a Trello board using the divide and conquer problem-solving approach. Below is an example of how my board looked like.
Get news data from some source.
Content is king, is saying any content creator worth his salt knows. I would not pretend to be a content creator, I just write code. From the get-go, I knew I would not want to use my already limited time researching and getting news, there were already established bloggers, and news agencies working around the clock to do that, there is no way I would be able to deliver news faster than were, neither would I write more compelling articles, since they had professional copywriters and editorial teams in terms of churning out information they surpassed me in every way. However my competitive advantage was that I knew web scrapping, I understood how websites worked and I had roughly 3 hours of free time after work since work from home basically eliminated the 2 hours I spent commuting to and from work. I decided to scrap a bunch of sites, which I think we're doing a great job creating content but did a lousy job with how the information was presented (it’s a shame some of the brightest minds aren’t, working on getting us to space faster, curing cancer or building next-gen AI, but are rather working on ads, it hurts my head thinking about it). I came up with 15 sites I wanted to scrape data from then got to work.
Data Scraping & Collection
Scraping data from these websites was quite easy since I had some previous experience scraping data from Tonaton.com (a peer to peer selling e-com business, at a point, I had a copy of all their data, but without a clear use for the data, the project was discontinued). I scraped the data using python, the Beautiful Soup library was used for static websites, selenium web drivers were used for websites that loaded data asynchronously using Ajax or some other means, for the odd cases where the data source didn’t have HTML, for example scraping a news API, I used python to request the endpoint I needed and then map the data to various entries in my database. The database was a No-SQL database just because I thought the structure of the collections might be changed at any time, hence the flexibility of a No-SQL database would make life easier, the No-SQL database was hosted on Mongo Atlas.
Rest API
An API was needed to connect the database with whatever interface the user would want to get news from, I designed the REST API using nodeJS. After some brainstorming, I realised that every news application should have four primary routes which are
- A route to get all news in the database
- A route to return all news in a given category
- A route to return news depending on a unique identifier
- A route to return news according to a search term
However since then I have come up with other possible routes, I realised would be useful, for other developers who would want to use my news API (after all, what is data if you don’t get to share it 😜).
User Interface
Now I had created all the underlining structures that any traditional news website would have, and it was time to create the user interface through which users would interact with the data I had collected. I decided to first create the React web interface, this step took me roughly a month to get a working version of the frontend running, because I had to learn the framework whilst building it.
Deployment
The application was deployed using a very simplified microservice architecture, where all services i.e the scraper, the database, API and frontend were hosted in various Heroku processes. This is illustrated in the diagram below.
Conclusion
The app has been deployed on Heroku and can be accessed by visiting https://newsfoldr.herokuapp.com/.
