Filtering Large Datasets

What is it?

Here are a series of examples to show you how to work with large sets of data.

Why should I learn it?

These examples will show you how you filter large datasets with user interaction.

What can I do with it?

Feel free to modify and customize any of these scripts for your own purposes.

How long will it take?

You already know all the JavaScript you need to be able to apply these concepts. These examples will allow you to apply them to larger datasets.

Getting Started

This series of exercises and examples will allow you to see how to apply interactive techniques when working with large datasets. This will allow you to read data and provide ways for users to explore it.

The data used in these exercises was found on the City of Austin Data Portal, specifically the Austin Fire Department's Fire Incidents for 2018. The data itself can be found in the file data.json. Download it and place it in the folder where you save the following examples. The data itself started as the spreadsheet from the data portal. Useful columns for incident number, month, battalion, problem, status, priority, latitude and longitude were selected. A CSV to JSON converter was used to get the file into the appropriate format. If your data includes apostrophes or quotes in a text item, those lines must be escaped in order for them to be read correctly by the script.

\' or \"

Note the use of the script tag to link to the dataset.

<script src="data.json"></script>

Read JSON Without Filter

Filter with a Dropdown

This example allows the user to choose a selection from a dropdown and see the incidents associated with that particular month. Note that in all these examples, the script begins by clearing out the DOM element on each form action to prevent continued appending of the results.

Filter with a Dropdown

Filter with Text Input

This example shows how to create a text search within a column of data. The script searches for a text input and matches it with a substring in the "Problem" column, starting at the beginning of the string.

Filter with Text Input

Filter Within String

This example looks to match a text input within a string. This example uses the JavaScript includes() method to identify whether a string is included in an item.

Filter Within String

Filter With Map

Since the Google Map now requires a credit card to get an API key, I no longer use this option in class. But if you want to see how it works, you can try it. When you publish, it will have a watermark on it.

Now that you can filter the results, you may need to use them in a chart or map. The AFD Fire Incidents data includes latitude and longitude for the addresses, and you might want to show these on a map based on the selected month.

Recall that the data for a Google Chart must be in an Array Table rather than JSON format. This can also be accomplished by using theCSV to JSON converter. The data in this format can be found in the file data2.json

If your map does not show when you upload to your domain, it is probably because you need a Google Maps API. You need to create one by going to the Google API Console and getting an API key. Set up a project for your site.

Filter with Map

Now You Try

Find the complete dataset of Covid data in Texas by region in JSON format.

  1. Copy the data into a file named data.json.
  2. Add a dropdown to select regions 1-11
  3. Write the script to match the region selected with the region in the code and displays the associated counties in a table
  4. Have the script also provide a count of the number of counties returned by the query

Pay careful attention to the names of the counties in the json

Filter Data Exercise

Get started with the code:

Put the data below in a file named data.json

Moving On

All these exercises demonstrate practical use of the features we have discussed, this time working with a large dataset. Apply these techniques to manipulate your own datasets.