Today, someone walked into an office that no longer belongs to them and, besides giving me an excellent excuse not to work, indirectly presented me with a rather interesting challenge: programmatically extracting highlights and notes created with the Google Scholar PDF Reader extension. At first, the task seemed fairly trivial. A few simple JavaScript functions executed directly from the browser console would probably have been enough. And indeed, within a few minutes I had a simple script capable of extracting most of the data I was interested in from the modal window that appears when clicking the “Highlights” menu. The problem was that another request was soon added—one that sounded harmless at first:

“Would it be possible to make the highlights include the entire sentence, just to provide a bit more context?”

The added difficulty came from the fact that there is no direct way to retrieve this information. The text must therefore be extracted directly from the original PDF, a process whose potential complications I was completely unaware of.

By that point, however, my curiosity had been sufficiently piqued, so I decided to accept the challenge.

This post is therefore a report on how I approached the problem, the obstacles I encountered, and the reasons that led me to favor one methodology over another.

Design

Glossary

Highlight: A portion of text selected by the user and saved through the Google Scholar PDF Reader extension. It may have different colors.
Note: A textual comment associated with a highlight. It may be empty.
Annotation: A highlight together with its associated note. Each annotation belongs to a PDF document.
PDF Document: A PDF file viewed through Google Scholar PDF Reader. Each document may have multiple annotations.
Page: A portion of a PDF document. Each annotation is associated with a page.
Sentence: A portion of text beginning and ending with a period, question mark, or exclamation mark. Each annotation is associated with a sentence in the PDF document.

Requirements

The goal of this project is to create a tool that satisfies the following requirements:

It must be possible to extract annotations from Google Scholar programmatically.
Each note must be associated with the highlight that generated it.
Each annotation must be associated with the page of the PDF document from which it originated.
Each annotation must be associated with the smallest group of sentences that contains it.
The tool’s output must be available:
- in JSON format;
- in CSV format;
- in Markdown format.
The tool must be easy to use and require no particular technical knowledge.

Implementation

Project Structure

After cloning the repository, the project structure will look like this:

.
├── dist                 # Generated by Rolldown
├── src                  # Source code
│   ├── background       # Background script entry point
│   └── content.ts       # Content script entry point
├── assets               # Static resources
│   └── icons
├── _locales             # Translations
│   ├── en
│   └── it
├── package.json
├── rollup.config.ts     # Rolldown configuration
└── manifest.json        # Extension configuration

In the following sections, we will examine the role of each file and folder in detail.

Tool Type

The first decision concerned the type of tool to implement.

At first glance, there are several possibilities:

a small JavaScript script executed directly in the browser console;
a simple command-line application, written in Python or Node.js, capable of controlling the browser through tools such as Selenium or Puppeteer;
a Chrome extension.

Among these options, the third seemed the most convenient.

Since users already need the Google Scholar PDF Reader extension installed in order to create annotations, installing an additional extension falls within their skill set.

Furthermore, this approach avoids dealing with security details (session cookies, authentication, etc.) that would be necessary with the second option, and it does not carry the same sense of “risk” or require even the minimal coding familiarity associated with the first option.

How the Extension Works

Chrome extensions are quite flexible and can execute code in different contexts depending on the requirements.

In our case, two categories of scripts are relevant:

Content scripts
Background scripts

Content scripts run within the web page currently being viewed by the user.

They can interact with the page’s DOM and generally have access to all the capabilities of a script imported directly into HTML.

Background scripts, on the other hand, run in a separate context.

They are used to manage extension state, communicate with content scripts, and do not block the execution of the web page.

The need for content scripts should be obvious: we need to read the page state (to access the user’s annotated articles) and modify the page content (to add commands).

The background script allows us to use the pdf.js library, which we use to enrich annotations with the context of the complete sentence.

To do this, we must manually download the original PDF, parse it correctly, locate the highlighted text, and return the complete sentence.

Most steps in this pipeline can fail quite easily—the PDF may not be available, the extracted text may be unreadable, or the sentence may be split across too many separate sections—but achieving something that works a good percentage of the time would already be a success.

The two scripts communicate through an asynchronous messaging system: the content script sends a request and receives a response after the background script has completed processing.

Loading diagram...

Language Choice

As for the programming language, the choice was fairly straightforward: TypeScript.

Since everything must run in a browser environment, the code ultimately needs to be transpiled to JavaScript and preferably minified as well—both tasks that Rolldown handles excellently with very little configuration.

In practice, it is enough to install Rollup, create a configuration file, and add a build script to package.json in order to obtain a distributable bundle.

# Install Rollup and Chrome TypeScript definitions
npm install --save-dev rollup chrome-types

// rollup.config.ts
import { defineConfig } from "rolldown";

export default defineConfig({
  input: "src/content.ts",
  output: {
    file: "dist/content.js",
    minify: true,
  },
});

{
  "scripts": {
    "build": "rollup -c",
    "watch": "rollup -c -w"
  }
}

Manifest

A Chrome extension consists of a collection of files, among which the most important is manifest.json, which contains the information required by the browser to install and manage the extension.

Most fields are self-explanatory, but for more information it is always advisable to consult the official documentation.

The only noteworthy aspect is the use of the localized strings __MSG_extension_name__ and __MSG_extension_description__, which allow the name and description fields to be translated according to the user’s language.

To provide these values, a messages.json file must be created inside _locales/<lang>/, where <lang> is the language code (en for English, it for Italian, etc.).

{
  "$schema": "https://json.schemastore.org/chrome-manifest",
  "manifest_version": 3,
  "name": "__MSG_extension_name__",
  "version": "0.0.1",
  "description": "__MSG_extension_description__",
  "icons": {
    "48": "./assets/icons/icon-48.png",
    "128": "./assets/icons/icon-128.png"
  },
  "default_locale": "en"
}

{
  "extension_name": {
    "message": "Google Scholar Annotation Exporter",
    "description": "The name of the extension"
  },
  "extension_description": {
    "message": "Export your Google Scholar PDF Reader annotations (highlights and notes) to a JSON, CSV, or Markdown file.",
    "description": "The description of the extension"
  }
}

Templating

There are quite a few JavaScript templating engines available, although many are understandably aimed primarily at generating HTML.

After a brief search, I decided to use Handlebars.js.

It has a syntax similar to what I am accustomed to, is easy to use, widely adopted, and well documented.

Limitations

In its current state, Google Scholar Annotation Exporter has several limitations, some of which appear difficult to overcome.

Limitation	Cause	Possible Solution
Google rate limiting	Google Scholar usage limits	Use an incognito browser session, change your connection, or use a different computer
Failure to extract context	PDF unavailable or irregular PDF structure	Ensure the URL is valid; prefer PDFs with easily parseable structures
Non-customizable output format	The tool exports in a predefined format	See TODO #2
CORS-protected PDF	Many PDFs enforce cross-origin request policies	See TODO #3

TODO

Allow users to override the PDF links provided by Google Scholar with alternative URLs supplied by them.
- In practice, this would be a simple URL => URL mapping.
- Whenever a URL about to be used exists as a key in the map, its associated value is used instead.
Allow users to specify a custom export format.
- A templating system such as Handlebars.js appears very promising for this purpose.
Create a system capable of bypassing CORS restrictions.
- This might involve creating an external proxy component, using a service such as CORS Anywhere, or relying on a service such as PDF.co to download PDFs.
Allow custom logic during PDF parsing.
- A particularly delicate and somewhat controversial feature from a security perspective, but potentially useful for advanced users.
Add additional metadata and information.

Google Scholar annotation exporter