Tentopolis > Google Scholar annotation exporter

Today, someone walked into an office that no longer belongs to them and, besides giving me an excellent excuse not to work, indirectly presented me with a rather interesting challenge: programmatically extracting highlights and notes created with the Google Scholar PDF Reader extension. At first, the task seemed fairly trivial. A few simple JavaScript functions executed directly from the browser console would probably have been enough. And indeed, within a few minutes I had a simple script capable of extracting most of the data I was interested in from the modal window that appears when clicking the “Highlights” menu. Another request was soon added, one that sounded harmless at first: “Would it be possible to make the highlights include the entire sentence, just to provide a bit more context?” The added difficulty came from the fact that there is no direct way to retrieve this information. The text must therefore be extracted directly from the original PDF, a process whose potential complications I was completely unaware of. By that point, however, my curiosity had been sufficiently piqued, so I decided to accept the challenge. This post is therefore a report on how I approached the problem, the obstacles I encountered, and the reasons that led me to favor one methodology over another.

Result

The final result is a Chrome extension called Google Scholar Annotation Exporter. The extension is open source and available on GitHub.

Design

Glossary

Highlight: A portion of text selected by the user and saved through the Google Scholar PDF Reader extension. It may have different colors.
Note: A textual comment associated with a highlight. It may be empty.
Annotation: A highlight together with its associated note. Each annotation belongs to a PDF document.
PDF Document: A PDF file viewed through Google Scholar PDF Reader. Each document may have multiple annotations.
Page: A portion of a PDF document. Each annotation is associated with a page.
Sentence: A portion of text beginning and ending with a period, question mark, or exclamation mark. Each annotation is associated with a sentence in the PDF document.

Requirements

The goal of this project is to create a tool that satisfies the following requirements:

It must be possible to extract annotations from Google Scholar programmatically.
Each note must be associated with the highlight that generated it.
Each annotation must be associated with the page of the PDF document from which it originated.
Each annotation must be associated with the smallest group of sentences that contains it.
The tool’s output must be available:
- in JSON format;
- in CSV format;
- in Markdown format.
The tool must be easy to use and require no particular technical knowledge.

Implementation

Project Structure

After cloning the repository, the project structure will look like this:

.
├── _locales             # Translations
│   ├── en
│   └── it
├── assets               # Static resources
│   ├── html
│   ├── style
│   └── icons
├── src                  # Source code
│   ├── background       # Background script entry point
│   └── content.ts       # Content script entry point
├── dist                 # Generated by Rolldown
├── package.json
├── rollup.config.ts     # Rolldown configuration
└── manifest.json        # Extension configuration

Tool Type

The first decision concerned the type of tool to implement.

At first glance, there are several possibilities:

a small JavaScript script executed directly in the browser console;
a simple command-line application, written in Python or Node.js, capable of controlling the browser through tools such as Selenium or Puppeteer;
a Chrome extension.

Among these options, the third seemed the most convenient: users already need the Google Scholar PDF Reader extension installed in order to create annotations, installing an additional extension falls within their skill set. Moreover, this approach avoids dealing with security details (session cookies, authentication, etc.) that would be necessary with the second option, and it does not carry the same sense of “risk” or require even the minimal coding familiarity associated with the first option.

How does the extension work

Chrome extensions are quite flexible and can execute code in different contexts depending on the requirements. In our case, two categories of scripts are relevant:

Content scripts, which run within the web page currently being viewed by the user. They can interact with the page’s DOM and generally have access to all the capabilities of a script imported directly into HTML.
Background scripts, which run in a separate context and are used to manage extension state, communicate with content scripts, and do not block the execution of the web page.

The need for content scripts should be obvious: we need to read the page state (to access the user’s annotated articles) and modify the page content (to add commands). The background script allows us to use the pdf.js library, which we use to enrich annotations with the context of the complete sentence. To do this, we must manually download the original PDF, parse it correctly, locate the highlighted text, and return the complete sentence. Most steps in this pipeline can fail quite easily (the PDF may not be available, the extracted text may be unreadable, or the sentence may be split across too many separate sections), but achieving something that works a good percentage of the time would already be a success.

The two scripts communicate through an asynchronous messaging system: the content script sends a request and receives a response after the background script has completed processing.

Loading diagram...

Language Choice

As for the programming language, the choice was fairly straightforward: TypeScript. Since everything must run in a browser environment, the code ultimately needs to be transpiled to JavaScript and preferably minified as well—both tasks that Rolldown handles excellently with very little configuration. In practice, it is enough to install Rollup, create a configuration file, and add a build script to package.json in order to obtain a distributable bundle.

# Install Rollup and Chrome TypeScript definitions
npm install --save-dev rollup chrome-types

// rollup.config.ts
import { defineConfig } from "rolldown";

export default defineConfig({
  input: "src/content.ts",
  output: {
    file: "dist/content.js",
    minify: true,
  },
});

{
  "scripts": {
    "build": "rollup -c",
    "watch": "rollup -c -w"
  }
}

Manifest

A Chrome extension consists of a collection of files, among which the most important is manifest.json, which contains the information required by the browser to install and manage the extension. Most fields are self-explanatory, but for more information it is always advisable to consult the official documentation. The only noteworthy aspect is the use of the localized strings __MSG_extension_name__ and __MSG_extension_description__, which allow the name and description fields to be translated according to the user’s language. To provide these values, a messages.json file must be created inside _locales/<lang>/, where <lang> is the language code (en for English, it for Italian, etc.).

{
  "$schema": "https://json.schemastore.org/chrome-manifest",
  "manifest_version": 3,
  "name": "__MSG_extension_name__",
  "version": "0.0.1",
  "description": "__MSG_extension_description__",
  "icons": {
    "48": "./assets/icons/icon-48.png",
    "128": "./assets/icons/icon-128.png"
  },
  "default_locale": "en",
  "content_scripts": [
    {
      "js": ["dist/content.js"],
      "matches": [
        "https://developer.chrome.com/docs/extensions/*",
        "https://developer.chrome.com/docs/webstore/*"
      ]
    }
  ]
}

.
├── _locales
│   ├── en
│   │   └── messages.json
│   └── it
│       └── messages.json
└── ...

// _locales/en/messages.json
{
  "extension_name": {
    "message": "Google Scholar Annotation Exporter",
    "description": "The name of the extension"
  },
  "extension_description": {
    "message": "Export your Google Scholar PDF Reader annotations.",
    "description": "The description of the extension"
  }
}

Templating

There are quite a few JavaScript templating engines available, although many are understandably aimed primarily at generating HTML. After a brief search, I decided to use Handlebars.js: the syntax is similar to what I am accustomed to, it is easy to use, widely adopted, and well documented.

Using a template engine introduces challenges I hadn’t considered. Chrome is extremely stringent when it comes to security: all scripts are associated with a minimal Content Security Policy that doesn’t allow inline code execution via eval(), new Function(), or similar.
The first solution I found was to precompile the templates, transforming them into standard JavaScript files to include in the extension. Once integrated into the bundle’s build workflow, everything works fine, and obviously efficiency improves significantly. However, this approach has a major drawback: you can’t modify or add new templates without recompiling the extension, and I wanted users to be able to use custom templates without having to mess with the source code. There’s a known workaround, which is to relegate these dangerous operations to a script inside a sandboxed iframe. The complication is that you can’t add this script anywhere: first, the HTML page must be declared in the manifest.json, and second, the iframe can only be inserted into pages belonging to the extension’s domain, which rules out doing so directly in the Google Scholar page, via the content script. Fortunately, I found a solution to this problem too, inspired by this article: using the Offscreen API, you can create an “invisible” HTML document. While it’s not considered a sandbox, we can take this a step further and insert the sandbox iframe, which allows us to execute inline code without security issues. The result is as expected, although the cascade of calls is certainly not the most elegant.

Loading diagram...

Using RollDown as a build system

RollDown already has a fairly advanced plugin system and features, especially considering it’s a relatively young project. The only features I had to add were:

the ability to include raw files (such as HTML files) in the final bundle;
compile HBS templates into JS files;
track updates to HTML and HBS template files, so you can update the extension’s status without having to restart the watch process;

Nothing particularly complex. The result was the following configuration file:

import { defineConfig } from "rolldown";
import Raw from "unplugin-raw/rolldown";
import build from "./scripts/build";
import path from "path";
import fs from "fs";

export default defineConfig(config => {
  const minify = config.watch ? false : true;
  build();
  return [
    {
      input: "src/content.ts",
      output: {
        dir: "dist",
        minify: minify,
      },
      plugins: [
        Raw(),
        {
          name: "handlebars-and-html-watcher",
          buildStart() {
            // Add HTML and HBS files to the list of files to watch
            fs.globSync(path.join(__dirname, "/src/**/*.{html,hbs}")).forEach(
              this.addWatchFile.bind(this)
            );
          },
          // When a watched file changes, rebuild the templates
          watchChange(id) {
            if (id.endsWith(".hbs")) build();
          },
        },
      ],
    },
    {
      input: "src/background.ts",
      output: {
        dir: "dist",
        minify: minify,
      },
      plugins: [Raw()],
    },
    // ...
  ];
});

where Raw() is a plugin that allows you to include raw files in the bundle, and build() is a function that compiles HBS templates into JS files and copies HTML files into the dist/ folder.

import Handlebars from "handlebars";
import fs from "fs";

export default function precompileTemplates() {
  // List all files in the src/templates directory that end with .hbs
  fs.readdirSync("src/templates")
    .filter(file => file.endsWith(".hbs"))
    .map(file => file.replace(".hbs", ""))
    .forEach(f => {
      // Precompile each HBS file and write the result to a .hbs.js file
      const template = fs.readFileSync(`src/templates/${f}.hbs`, "utf-8");
      const specification = Handlebars.precompile(template, {
        destName: `${f}.hbs.js`,
        srcName: `${f}.hbs`,
        knownHelpersOnly: true,
        knownHelpers: {
          escapeDoubleQuotes: true,
          escapePipe: true,
          escapeTex: true,
        },
      }) as unknown as { code: string };
      fs.writeFileSync(
        `src/templates/${f}.hbs.js`,
        `export default ${specification.code};`
      );
    });
}

Limitations

In its current state, Google Scholar Annotation Exporter has several limitations, some of which appear difficult to overcome.

Limitation	Cause	Possible Solution
Google rate limiting	Google Scholar usage limits	Use an incognito browser session, change your connection, or use a different computer
Failure to extract context	PDF unavailable or irregular PDF structure	Ensure the URL is valid; prefer PDFs with easily parseable structures TODO 1
CORS-protected PDF	Many PDFs enforce cross-origin request policies	See TODO 2

TODO

Allow users to override the PDF links provided by Google Scholar with alternative URLs supplied by them. In practice, this would be a simple URL => URL mapping. Whenever a URL about to be used exists as a key in the map, its associated value is used instead.
Create a system capable of bypassing CORS restrictions. This might involve creating an external proxy component, using a service such as CORS Anywhere, or relying on a service such as PDF.co to download PDFs.
Allow custom logic during PDF parsing. A particularly delicate and somewhat controversial feature from a security perspective, but potentially useful for advanced users.
Add additional export formats, such as BibTeX or EndNote.