Skip to main content

Config.LinkAuto

Note (2026-07): This module has been permanently removed from the codebase (commit fb6938236, "LinkAuto: permanently remove, due to pervasive slowdown and decreasing utility"). The main LinkAuto.hs module survives only in build/old/. This page describes the configuration as it existed before removal.

Archived configuration data for the old auto-linking system: regex patterns and URL mappings

Pathbuild/old/LinkAuto.hs
LanguageHaskell
Lines1311
Sourcebuild/old/LinkAuto.hs
at 406d3e423

Read this when Use this page when tracing the Haskell build pipeline, generators, metadata code, or backend utility behavior around Config.LinkAuto.

Overview​

Config.LinkAuto was a pure data module that defined the dictionary of regular expression patterns and their corresponding URLs for gwern.net's former auto-linking system. In the current source tree, the archived definitions are retained inside build/old/LinkAuto.hs for reference rather than participating in the active build.

The module contains approximately 1,000 regex→URL pairs covering technical terms (AI/ML model names like "GPT-3", "BERT", "StyleGAN"), statistical concepts ("GWAS", "MCMC", "PGS"), notable people (researchers, authors, historical figures), media (anime, books, websites), and scientific terminology (genetics, psychology, neuroscience). Each pattern uses Haskell's regex-tdfa syntax with alternations and character classes to match multiple surface forms of the same concept.

The patterns are sorted by length in descending order at compile time (customSorted) so that longer, more specific matches take priority over shorter generic ones—"BigGAN" matches before "GAN", and "reinforcement learning" matches before "learning". This design prevents greedy short patterns from capturing text that should be linked to more specific definitions.

Public API​

custom :: [(T.Text, T.Text)]​

The raw list of (regex, URL) pairs defining auto-link patterns.

Archived use: LinkAuto.customDefinitions Calls: (pure data)

customSorted :: [(T.Text, T.Text)]​

The same list as custom, sorted by descending pattern length for priority matching.

Archived use: LinkAuto.customDefinitions Calls: Data.List.sortBy, Data.Text.length

linkAutoTests :: [([Inline], Pandoc)]​

Test cases pairing input Inline fragments with expected Pandoc output after auto-linking.

Archived use: LinkAuto.linkAutoTest Calls: (pure data using Pandoc constructors)

Internal Architecture​

Pattern Structure​

Each entry is a 2-tuple of (regex, url):

("(GPT-3|Brown[  ]et[  ]al[  ]2020)", "https://arxiv.org/abs/2005.14165#openai")

Patterns use regex-tdfa syntax:

  • (A|B|C) — alternation for synonyms/variants
  • [Xx] — case-insensitive character class
  • .? — optional character (often for plurals)
  • [ -] — space or hyphen variants
  • \\. — escaped literal periods

URLs can be:

  • Absolute Wikipedia/arXiv/external URLs
  • Site-relative paths like /modafinil or /doc/...

Sorting Logic​

customSorted = sortBy (\a b -> compare (T.length $ fst b) (T.length $ fst a)) custom

Longest patterns first ensures specificity. Without this, "GAN" would match inside "BigGAN" before "BigGAN" gets a chance.

Test Data​

The linkAutoTests list contains regression tests as ([Inline], Pandoc) pairs:

  1. Input: Pandoc Inline fragments (Str, Link, Space, etc.)
  2. Expected: Complete Pandoc document after auto-linking

Tests verify:

  • Links inside existing Link nodes are NOT auto-linked (would create invalid nested links)
  • First occurrence only gets linked
  • Multiple distinct terms in same text all get their respective links
  • Case sensitivity behaves as specified
Key Patterns

Pattern Categories​

CategoryExamplesCount (approx)
ML/AI modelsGPT-3, BERT, StyleGAN, AlphaGo, MuZero~150
StatisticsGWAS, PGS, MCMC, Bayesian, regression~100
PeopleBorges, Feynman, Nietzsche, von Neumann~150
GeneticsSNP, QTL, heritability, epistasis~75
PsychologyBig Five, ADHD, schizophrenia~50
ComputingBitcoin, Tor, Emacs, Haskell~75
Anime/MediaEvangelion, Cowboy Bebop, Gainax~50
Misc conceptsEfficient market, Fermi problem~100+

Pattern Design Considerations​

1. Variant Coverage

("(ADHD|[Aa]ttention[ -][Dd]eficit [Hh]yperactivity [Dd]isorder)s?", "...")

Matches "ADHD", "attention-deficit hyperactivity disorder", "Attention Deficit Hyperactivity Disorder", etc.

2. Disambiguation by Specificity

("StyleGAN2-ADA.?|Karras et al 2020", "...2006.06676...")
("(StyleGANs?|CelebA[ -]HQ|FFHQ)", "...1812.04948...")
("StyleGAN2s?", "...1912.04958...")

Multiple StyleGAN entries, each mapping to the correct paper.

3. Citation Patterns

("(GPT-3|Brown[  ]et[  ]al[  ]2020)", "https://arxiv.org/abs/2005.14165#openai")

Matches both the common name and the academic citation format.

Configuration

Performance Warning​

The file contains a hard limit check in LinkAuto.hs:

if length C.customSorted > 1007 then error "..."

Too many patterns cause exponential slowdown. The comment in the config notes:

"From now on, delete at least one rewrite for every added rewrite."

Adding New Patterns​

  1. Add tuple to custom list
  2. Ensure pattern is unique enough (won't match inside other terms inappropriately)
  3. Use longest/most-specific form as the base
  4. Include common variants via alternation
  5. Consider removing an obsolete entry to stay under the limit

Integration Points​

Used By​

  • LinkAuto — Archived consumer that applied patterns to documents
  • Archived LinkAuto tests — Historical validation data retained with the old module

Validation​

The archived test path performed several checks:

  1. isUniqueAll — No duplicate regex patterns
  2. isURLAnyT — All URLs pass basic URL validation
  3. Regex compilation test — All patterns compile successfully

URL Formats​

The URLs use gwern.net conventions:

  • #openai, #deepmind, #google — Organization tag fragments for arXiv links
  • /doc/... — Local document paths
  • /modafinil, /nicotine — Short-URL essay references
  • Wikipedia/external links for general concepts
See Also