CSS JavaScript Netlify Project React Scraping Serverless

Project Summary: NHS Dentist Finder

Being able to build digital products is such an enabler.

This project was a great example of experiencing a problem or difficulty and, having enough knowledge to understand how a solution could work, being able to just go ahead and build it and fill in any gaps by learning along the way.

Check out the demo here (and the code here).

What is it and what does it do?

It searches the NHS website for the nearest dentists to you that are accepting adult NHS patients.

This is a solution to a problem I experienced when trying to find and register with a local dental practice. If you Google “find nhs dentist”, you’ll come across this result. I’m a fan of the modern NHS UI, but the actual UX of finding a dentist is poor. You have to scroll down and scan a list (see below) of 50+ entries, searching for the piece of text in each one that says “Is this dentist taking new NHS patients?”. It’s not the easiest or quickest thing when there’s hundreds to go through.

So I called NHS England’s Customer Contact Centre as they recommend, and was told that they’re aware the UX is crap so I should try a different website of theirs instead. They suggested this one.

“Great!”, I thought. It has clear “No” and “Yes” signs that are red and green respectively. Much easier.

However, when you realise that only 2 of the nearest 100 dental practices are accepting adult NHS patients I realised that, actually, this is still fairly shit UX. Just show me only the available dentists near me!

So I built this.

How was it built?

The UI was built using React from a Create React App starter and the scraping is handled by a serverless Netlify function that runs Puppeteer with Chromium. Other notable features include:

  • Use of React Hooks, such as useEffect for fetch calls
  • Navigator browser API for geolocation
  • A free Postcodes API for retrieving and validating location details

The visuals didn’t need to be complex, so I used plain CSS.

The way it works is:

  1. User supplies location details (postcode, latitude, longitude) via one of two methods;
    1. Geolocation – uses the Navigator browser API to get the user’s latitude and longitude, then sends those to the Postcodes API to retrieve the corresponding postcode and fills the <input> field for the user.
    2. Manual postcode entry – validates the user-entered postcode with the Postcodes API (is it a legit postcode?), if so it retrieves the corresponding latitude and longitude from the Postcodes API.
  2. User clicks “Find a dentist” and the location details are sent to a serverless Netlify function.
  3. The function starts up Puppeteer which loads a specified URL with the location details dynamically embedded and scrapes all available dentists details from the page, then returns them to the React app.
  4. The app displays the results, as well as two further options:
    1. Search for more – increments a pageNum variable which is sent along with the location details to the Netlify function so that the next page on the NHS website is scraped to bring back the next 50 dentists.
    2. Show unavailable – displays in a table the unavailable dental practices and those with unknown availability.

To date, I’ve spent 18 hours on the project (the longest I’ve spent on a project in recent months).

What did I learn?

I spent so much time on this because yes, it was a little larger than the previous projects, but mostly because it was more complex and introduced some features or services which I’ve not used before.

The learnings in this project were:

  • First time using Netlify Functions – I’ve used AWS Lambda serverless functions before and I’ve used Netlify before, but not Netlify’s serverless functions 😁
  • Practice with the browser’s Navigator API for getting geolocation data
  • Practice handling errors from API calls, invalid user input, etc
  • Practice showing “Loading…” messages while a background function is being run
  • First time using the Postcodes API – really useful, and free!
  • Practice chaining fetch calls, such as first validating a postcode and then retrieving data about the valid postcode
  • Practice debouncing in a useEffect hook
  • Practice with Puppeteer and headless Chromium

What’s next?

It’s a fully functional app at the moment, but there’s some cleaning up that could be done as well as features to be added, such as:

  • Handle cases not currently catered for, such as a user clicking “Find Dentist” without a postcode
  • Show locations on a map for users to understand the distance with context
  • Scrape more data (i.e. not just adult patient availability)
  • Cache the scraped data

Leave a Reply

Your email address will not be published.