Lesson 22: Automating Pages You Use

22.1 JavaScript on Pages You Don't Own

Everything so far has run on pages you wrote. But the browser will run your JavaScript on any page you're viewing, including sites you use every day. That opens up a useful kind of personal automation: pulling data out of a page, filling in a form you fill in every week, or hiding parts of a site you find distracting. The changes happen only in your browser, only for you.

There are three ways to do it, from quickest to most automatic:

• The Console and Snippets in the developer tools: run code by hand, on the page in front of you.
• Bookmarklets: a bookmark that runs code when you click it, on whatever page you're on.
• Userscripts: code that runs by itself every time you open a particular site, using a browser extension called Tampermonkey.

22.1.1 Before you automate a site

Running code on a page you're viewing is legal and normal; it's what the developer tools are for. But what you do with it matters:

• Read the site's terms of use. Some sites forbid automated access, or copying their data. Filling in your own form faster is almost always fine; collecting other people's data from a site usually isn't.
• Only act as yourself. Automate what you could do by hand, in your own account, at a human pace. Code that sends hundreds of requests, or gets around a login or a limit, can break rules or laws, and can harm the site.
• Be careful with personal data. A table you copy out of a site may include information about other people, and it needs the same care as the club's form responses.

22.2 Selecting Many Elements

document.querySelector returns the first element that matches a selector. document.querySelectorAll returns all of them, as a list you can loop over with an index, like an array:

<table id="harvests">
  <tr><th>Date</th><th>Bed</th><th>Crop</th><th>Kilograms</th></tr>
  <tr><td>2027-04-18</td><td>B2</td><td>Radish</td><td>1.2</td></tr>
  <tr><td>2027-05-13</td><td>B4</td><td>Spinach</td><td>2.4</td></tr>
  <tr><td>2027-05-15</td><td>B2</td><td>Lettuce</td><td>3.9</td></tr>
</table>
<p id="total"></p>

<script>
  const rows = document.querySelectorAll("#harvests tr")
  let totalKg = 0
  // start at 1 to skip the header row
  for (let i = 1; i < rows.length; i++) {
    const cells = rows[i].querySelectorAll("td")
    const kilograms = Number(cells[3].textContent)
    totalKg += kilograms
  }
  const total = document.querySelector("#total")
  total.textContent = `${rows.length - 1} harvests, ${totalKg.toFixed(1)} kg in all`
</script>

The selector "#harvests tr", with a space, means "every tr inside the element with id harvests." And rows[i].querySelectorAll("td") searches inside one row only: querySelectorAll can be called on any element, not just document. Reading a page's table is the same as reading a sheet's rows, with textContent in place of getValues, and every value arriving as text.

22.3 Trying Code in the Console

The quickest way to run code on a page is the developer tools' Console, which you met in Lesson 19. How Web Pages Work. Open it on any page (F12), type a line such as document.querySelectorAll("a").length, and press Enter: it tells you how many links the page has. The Console runs your code on that page, so document is that page.

For code longer than a line or two, use Snippets, saved scripts you can run on any page: in Chrome or Edge, open the Sources panel, choose Snippets (you may need to click >> to find it), click New snippet, write or paste your code, and press Ctrl+Enter (Cmd+Enter on a Mac) to run it. Snippets are saved in your browser, so they're a good place to keep small tools while you're trying them out.

22.4 A Bookmarklet That Exports a Table

Many sites show data in tables with no way to download it. A bookmarklet that turns the first table on any page into a CSV file, which opens in Google Sheets or Excel, would be handy for the club, for example for a table of plant spacings on a gardening site. Here's the request, in a new chat with your learner profile from Lesson 21. Asynchronous JavaScript:

Read. The assistant listed four new things first, which is exactly what your profile asks.

• The loops are the ones you just saw: every tr in the table, then every td or th in each row. The selector "td, th", with a comma, means "either kind of cell." Each cell's text is wrapped in quotes and separated by commas, and each row ends with \n.
• A CSV file (comma-separated values) is plain text: one line per row, with commas between the values. The quotes around each value let a value contain a comma, like "Bed 3, south." The assistant pointed out the one case it doesn't handle, a value that contains a quote mark, which would need to be written as two quote marks.
• The download takes three steps. A Blob holds the CSV text as file-like data. URL.createObjectURL gives it a temporary address. A link to that address with a download attribute makes the browser save the file instead of opening it, and click() clicks the link from code.
• (function() { ... })() is a function with no name, written in parentheses and called immediately with the () at the end. It keeps the bookmarklet's variables inside the function, so they don't clash with the page's own variables. It also lets the code use return to stop early when there's no table.

Here's the CSV-building part on a page, with the download replaced by showing the text, so you can see exactly what the file would contain:

<table>
  <tr><th>Crop</th><th>Spacing (inches)</th></tr>
  <tr><td>Lettuce</td><td>12</td></tr>
  <tr><td>Pepper</td><td>18</td></tr>
  <tr><td>Tomato</td><td>24</td></tr>
</table>
<pre id="csv"></pre>

<script>
  const table = document.querySelector("table")
  const rows = table.querySelectorAll("tr")
  let csvText = ""
  for (let i = 0; i < rows.length; i++) {
    const cells = rows[i].querySelectorAll("td, th")
    let csvRow = ""
    for (let j = 0; j < cells.length; j++) {
      const cellText = cells[j].textContent
      if (j > 0) {
        csvRow = csvRow + ","
      }
      csvRow = csvRow + '"' + cellText + '"'
    }
    csvText = csvText + csvRow + "\n"
  }
  document.querySelector("#csv").textContent = csvText
</script>

22.4.1 When the bookmarklet does nothing

Following the assistant's instructions exactly, you'd join the lines into one, put javascript: in front, and save it as a bookmark. Clicking it does nothing:

Both suggestions are worth knowing. Some browsers do remove javascript: from pasted text in certain places, as a defense against the scams described above. And joining lines without semicolons really does break code: the line breaks were doing the job of separating statements, and once they're gone, something else has to. That's the one place this book adds semicolons back.

But the explanation misses the biggest problem. The original code is full of comments, and a // comment runs to the end of the line. Joined into one line, the first // Find the first table on the page turns everything after it into a comment, so almost none of the code is left. Look at the assistant's one-line version: it quietly removed every comment. Its fix works, but partly for a reason it didn't mention. When you compare a fix with the original, look for changes the explanation doesn't account for.

So, for code that has to fit on one line: remove // comments (or use the /* ... */ kind, which ends where it says), and separate statements with semicolons. The assistant's one-line version does both.

22.5 Userscripts

A bookmarklet runs when you click it. A userscript runs by itself, every time you open a page it's meant for. Userscripts need a browser extension; the most popular is Tampermonkey, which is free for Chrome, Edge, Firefox and Safari. Install it from your browser's extension store. (Recent versions of Chrome also require turning on Allow User Scripts for Tampermonkey in the extension's details page; Tampermonkey's own site explains the current steps.)

Every week, someone signs the club up for shifts on the university's volunteer portal, typing the same three things into the same form. Here's a request for a userscript that fills it in, in a new chat with your learner profile. (The portal's address and field names are made up for this example.)

Read. The code is short, because the hard part is done by Tampermonkey. The comment block at the top is its metadata: @name is what Tampermonkey shows in its list, @match is the address the script runs on, and @run-at document-idle waits until the page has loaded. The code finds the three boxes and sets their value, the reverse of reading it in Lesson 20. Events and Interactivity.

Two things to know before relying on it:

• @match decides where your code runs. A pattern like https://volunteer.example.edu/*, with a *, would run it on every page of the site. Keep it as narrow as the job needs, because a userscript runs with the same access to the page as the site's own code.
• Some sites don't notice values set by code. Many modern sites keep their own record of what's been typed, updated by input events. Setting value changes the box, but not that record, so the site may treat the box as empty when you submit. If that happens, the fix is to send an input event after setting the value; your assistant can show you how, and it's a good question to ask if the form seems to ignore your script.

The assistant noted who can see the script, as your profile asks, but said nothing about the site's rules. A script that fills in your own form is almost certainly fine. Still, it's your job to check, and it's worth a new profile rule.

22.6 Your Learner Profile

22.7 Summary

The browser will run your JavaScript on any page you're viewing, which makes it a tool for personal automation: in the Console and Snippets by hand, as a bookmarklet when you click it, or as a userscript that runs by itself on the pages its @match pattern names. querySelectorAll returns every matching element, and reading a web page's table is much like reading a sheet. A bookmarklet's code has to fit on one line, so // comments must go and statements need semicolons. Automate only what you could do by hand, in your own account, and check the site's terms. Above all, never paste code you can't explain into the Console, because it runs as you. Next, you'll package this kind of code as a browser extension of your own.