Camelot vs pdfplumber for Grant Budget Tables

Decision guide for choosing Camelot lattice vs pdfplumber when extracting budget tables from grant PDFs: gridline dependence, parsing_report accuracy, Ghostscript deps, word-coordinate control, a select_extractor() router, and 2 CFR §200.302 alignment.

This guide is part of the PDF Grant Application Parsing section within the broader Data Ingestion & Grant Parsing Workflows framework, and it answers one narrow question: when a budget table lands on your desk inside a grant PDF, do you reach for Camelot or for pdfplumber?

Both libraries read the same born-digital text layer, and both can be wrong in ways that silently corrupt a budget figure feeding a regulated artifact. The mistake is treating this as a taste preference. It is a structural decision: Camelot’s lattice flavor keys off ruled gridlines and wins on federal templates that draw them; pdfplumber reconstructs tables from word coordinates and wins on borderless, whitespace-delimited layouts where Camelot has no lines to follow. Neither reads pixels, so a scan routes to OCR before either engine sees it. This guide builds a select_extractor() router that inspects the page and picks deterministically, plus a harness that runs both engines and diffs their output before you trust either.

When to Use This Approach

Use this decision framework when all three of the following hold:

  • The PDF is born-digital, not scanned. Both engines read an embedded text layer and its coordinate map; neither reads a rasterized image. A scanned application has to be rasterized and OCR’d first — that path belongs to the OCR fallback for scanned grant applications with Tesseract, and the text-layer probe in Step 2 is the gate that enforces it.
  • You have a mixed funder corpus. A NIH modular budget and an NSF cumulative budget draw ruled grids; a private-foundation line-item sheet or a narrative budget justification often has no borders at all. A single hardcoded engine mis-extracts half the corpus. The router picks per document based on measured page geometry, not a global assumption.
  • The output reconciles to a regulated record. Extracted figures land in 2 CFR §200.302 financial-management records, so a column shifted by one or a merged cell split in two is a compliance event. The router records which engine produced which numbers so an auditor can replay the choice.

Currency normalization, indirect-cost allocation, and canonical column naming are explicitly out of scope here — those belong to Field Mapping & Normalization. This page decides the extraction engine and hands off a raw DataFrame plus a decision record; it does not clean the values.

Decision tree routing a grant PDF to Camelot lattice, pdfplumber, or OCR A grant budget page enters a two-question decision tree. The first gate asks whether an embedded text layer is present: if not, the page routes to OCR preprocessing because neither Camelot nor pdfplumber reads pixels. If text is present, the second gate asks whether ruled gridlines are detected on the page. When gridlines are present, the page routes to Camelot lattice, chosen because it keys off ruled cell borders and reports a parsing_report accuracy score. When gridlines are absent, the page routes to pdfplumber, chosen because it reconstructs borderless and whitespace-delimited tables from word coordinates and gives finer positional control. Two questions decide the engine Text layer, then gridlines — each page routes to exactly one extractor, and the choice is recorded. no text gridlines no gridlines Grant budget page one PDF page text layer? ruled gridlines? OCR preprocessing no pixels to either engine Camelot lattice ruled cells · parsing_report accuracy pdfplumber word coords · borderless tables

Step-by-Step Implementation

The reference implementation targets Python 3.10+. Camelot’s lattice flavor shells out to Ghostscript and depends on OpenCV, so it carries system-level baggage that pdfplumber — pure Python over pdfminer.six — does not. Install both plus pypdf for the text-layer probe:

bash
pip install "pdfplumber==0.11.4" "camelot-py[cv]==0.11.0" "pypdf==4.2.0" "pandas==2.2.2"
# Camelot lattice additionally requires Ghostscript at the system level:
#   Debian/Ubuntu:  apt-get install ghostscript
#   macOS:          brew install ghostscript
# pdfplumber has no system dependency.

Step 1: Compare the two engines across the dimensions that decide

Before writing a router, internalize where each engine actually wins. The comparison below is the decision surface — the router in Step 3 is just this table expressed in code.

Dimension Camelot (lattice) pdfplumber
Gridline dependence Requires ruled cell borders; lattice finds no table without them None; reconstructs tables from word/character coordinates, so borderless layouts work
Best-fit budget table NIH/NSF federal templates with drawn grids Foundation line-item sheets, narrative justification tables, whitespace-delimited columns
Accuracy reporting Yes — table.parsing_report["accuracy"], a 0–100 confidence score None; you validate structure yourself against expected column count
System dependencies Ghostscript + OpenCV ([cv] extra) Pure Python over pdfminer.six; nothing to apt-get
Merged-cell handling Detects spanning cells from the grid geometry directly No native span model; merges infer from coordinate gaps and need manual repair — see reconciling merged cells in funder budget workbooks
Coordinate control Coarse; you get cells, not per-word positions Fine; every word carries x0, x1, top, bottom for custom column inference
Memory profile Holds full coordinate map + Ghostscript raster per call; heavier Lighter per page; streams pages via pdfminer
Speed Slower on grid pages (Ghostscript raster round-trip) Faster on typical pages; no external process

Step 2: Probe the page for a text layer and ruled gridlines

The router’s two inputs are booleans: does the page carry embedded text, and does it carry ruled lines? Compute both from pdfplumber’s own page model — it exposes extract_text() and a lines/edges list — so a single open answers both questions. 2 CFR §200.302 requires records to be traceable, so log the probe result that drives the routing decision.

python
import logging
from dataclasses import dataclass
from pathlib import Path
import pdfplumber

AUDIT_LOGGER = logging.getLogger("grant_extraction.routing")
AUDIT_LOGGER.setLevel(logging.INFO)
_handler = logging.StreamHandler()
_handler.setFormatter(logging.Formatter("%(asctime)s | %(levelname)s | %(message)s"))
AUDIT_LOGGER.addHandler(_handler)

RULED_LINE_THRESHOLD: int = 4  # min. horizontal+vertical rules to call a grid


@dataclass(frozen=True)
class PageGeometry:
    """Immutable probe result that drives the routing decision."""
    has_text_layer: bool
    ruled_line_count: int

    @property
    def has_gridlines(self) -> bool:
        return self.ruled_line_count >= RULED_LINE_THRESHOLD


def probe_page_geometry(pdf_path: Path, page_index: int = 0) -> PageGeometry:
    """Inspect a single page for an embedded text layer and ruled lines."""
    with pdfplumber.open(str(pdf_path)) as pdf:
        page = pdf.pages[page_index]
        text = page.extract_text() or ""
        rules = len(page.lines) + len(page.edges)
        geometry = PageGeometry(
            has_text_layer=len(text.strip()) > 50,
            ruled_line_count=rules,
        )
    AUDIT_LOGGER.info(
        "Probe | page=%d | text=%s | rules=%d | grid=%s",
        page_index, geometry.has_text_layer, rules, geometry.has_gridlines,
    )
    return geometry

The > 50 character floor separates a born-digital page (hundreds of characters) from a scan (zero); the RULED_LINE_THRESHOLD of 4 requires at least a couple of horizontal plus vertical rules before calling a page a grid, which avoids mistaking a single header underline for a table border.

Step 3: Route to an engine with a deterministic default

The router turns the two probe booleans into an engine name. The logic mirrors the decision tree exactly: no text layer routes to OCR; text with gridlines routes to Camelot lattice; text without gridlines routes to pdfplumber. The deterministic default matters — when in doubt (text present, gridline count ambiguous), fall to pdfplumber, because it degrades to a best-effort word-coordinate table rather than the empty list Camelot returns on a missing grid.

python
from enum import Enum
from typing import NamedTuple


class Extractor(str, Enum):
    CAMELOT_LATTICE = "camelot_lattice"
    PDFPLUMBER = "pdfplumber"
    OCR = "ocr"


class RoutingDecision(NamedTuple):
    extractor: Extractor
    reason: str
    geometry: PageGeometry


def select_extractor(pdf_path: Path, page_index: int = 0) -> RoutingDecision:
    """Inspect a page and pick the extraction engine, defaulting to pdfplumber."""
    geometry = probe_page_geometry(pdf_path, page_index)

    if not geometry.has_text_layer:
        decision = RoutingDecision(
            Extractor.OCR,
            "No embedded text layer; neither engine reads pixels. Route to OCR.",
            geometry,
        )
    elif geometry.has_gridlines:
        decision = RoutingDecision(
            Extractor.CAMELOT_LATTICE,
            f"Ruled grid detected ({geometry.ruled_line_count} rules); "
            "lattice keys off cell borders and reports parsing_report accuracy.",
            geometry,
        )
    else:
        # Deterministic default: borderless / ambiguous pages go to pdfplumber.
        decision = RoutingDecision(
            Extractor.PDFPLUMBER,
            "No ruled grid; pdfplumber reconstructs the table from word coordinates.",
            geometry,
        )

    AUDIT_LOGGER.info("Route | engine=%s | %s", decision.extractor.value, decision.reason)
    return decision

Returning a RoutingDecision (engine + human-readable reason + the geometry it was based on) rather than a bare string gives the audit trail a replayable record: an auditor can see not just which engine ran but why, and re-probe the same page to confirm the choice.

Step 4: Run the chosen engine behind one contract

Each engine has a different call shape, so wrap both behind a single extract function that returns a DataFrame regardless of which ran. Camelot yields tables[0].df directly; pdfplumber yields a list of row lists that you frame yourself. An empty result from either is an explicit signal, never a swallowed exception.

python
from typing import Dict, List, Optional, Tuple
import pandas as pd
import camelot


def run_extractor(pdf_path: Path, decision: RoutingDecision) -> Tuple[pd.DataFrame, Dict[str, object]]:
    """Execute the routed engine and return a DataFrame plus a decision record."""
    record: Dict[str, object] = {
        "engine": decision.extractor.value,
        "routing_reason": decision.reason,
    }

    if decision.extractor is Extractor.OCR:
        raise RuntimeError("Page has no text layer. Route to OCR preprocessing, not this stage.")

    if decision.extractor is Extractor.CAMELOT_LATTICE:
        tables = camelot.read_pdf(str(pdf_path), flavor="lattice", pages="1", process_background=True)
        if len(tables) == 0:
            raise ValueError("Lattice found no table despite detected grid; re-route to pdfplumber.")
        df = tables[0].df
        record["accuracy_score"] = tables[0].parsing_report.get("accuracy", 0.0)
    else:  # pdfplumber
        with pdfplumber.open(str(pdf_path)) as pdf:
            rows: Optional[List[List[Optional[str]]]] = pdf.pages[0].extract_table()
        if not rows:
            raise ValueError("pdfplumber extracted no rows; page may need OCR or manual review.")
        df = pd.DataFrame(rows[1:], columns=rows[0])
        record["accuracy_score"] = None  # pdfplumber reports no confidence score

    record["rows_extracted"] = len(df)
    AUDIT_LOGGER.info("Extract | engine=%s | rows=%d", decision.extractor.value, len(df))
    return df, record

Note the asymmetry the record captures: Camelot fills accuracy_score from parsing_report, while pdfplumber records None because it offers no confidence metric — you validate its structure yourself in the harness below.

Step 5: Build a dual-engine validation harness

Before trusting the router on a new funder, run both engines on the same page and diff their output. Agreement on row count and cell content is strong evidence the extraction is correct; divergence flags a page for manual review. This harness is what you run once per funder template to calibrate the router, and again in CI whenever a funder changes their form.

python
from typing import Set


def compare_engines(pdf_path: Path) -> Dict[str, object]:
    """Run both engines on page 1 and report agreement for calibration/review."""
    result: Dict[str, object] = {"pdf_path": str(pdf_path)}

    camelot_df = pd.DataFrame()
    plumber_df = pd.DataFrame()

    try:
        camelot_df, _ = run_extractor(
            pdf_path, RoutingDecision(Extractor.CAMELOT_LATTICE, "harness", probe_page_geometry(pdf_path))
        )
    except (ValueError, RuntimeError) as exc:
        AUDIT_LOGGER.warning("Harness: Camelot failed | %s", exc)
        result["camelot_error"] = str(exc)

    try:
        plumber_df, _ = run_extractor(
            pdf_path, RoutingDecision(Extractor.PDFPLUMBER, "harness", probe_page_geometry(pdf_path))
        )
    except (ValueError, RuntimeError) as exc:
        AUDIT_LOGGER.warning("Harness: pdfplumber failed | %s", exc)
        result["pdfplumber_error"] = str(exc)

    result["camelot_rows"] = len(camelot_df)
    result["pdfplumber_rows"] = len(plumber_df)
    result["row_count_agrees"] = len(camelot_df) == len(plumber_df) and len(camelot_df) > 0

    if not result["row_count_agrees"]:
        AUDIT_LOGGER.warning(
            "Harness: engines disagree | camelot=%d rows | pdfplumber=%d rows | flag for review",
            len(camelot_df), len(plumber_df),
        )
    return result

Row-count agreement is the cheap first check; for high-value budgets, extend the harness to compare cell-by-cell on the numeric total column, since two engines can agree on shape while disagreeing on a single misread digit.

Verification

Confirm the router and harness behave deterministically with these checks:

  1. The text-layer gate routes scans to OCR. Feed a scanned-image PDF and assert select_extractor returns Extractor.OCR with a reason naming OCR, and that run_extractor raises RuntimeError before touching either engine.
  2. Gridlines route to Camelot, borderless routes to pdfplumber. Assert a NIH ruled-grid template returns Extractor.CAMELOT_LATTICE and a borderless foundation sheet returns Extractor.PDFPLUMBER, each with a reason referencing the geometry it measured.
  3. The default is deterministic. Assert that a text page with a rule count below RULED_LINE_THRESHOLD always falls to pdfplumber — same page, same engine, every run.
  4. The harness flags divergence. Run compare_engines on a page where the two engines split a merged cell differently and assert row_count_agrees is False and a WARNING audit line is emitted.

A routed run emits one INFO probe line, one INFO route line, and one INFO extract line, so the decision is fully reconstructable from logs. Ship those to a write-once tier so the trail supports the 2 CFR §200.302 records requirement.

python
decision = select_extractor(Path("NIH_R01_budget.pdf"))
assert decision.extractor in {Extractor.CAMELOT_LATTICE, Extractor.PDFPLUMBER, Extractor.OCR}

report = compare_engines(Path("NIH_R01_budget.pdf"))
assert "row_count_agrees" in report
assert report["camelot_rows"] >= 0 and report["pdfplumber_rows"] >= 0

Common Errors & Fixes

Error Cause Fix
camelot.read_pdf returns an empty TableList on a page that looked ruled Header underlines counted as a grid, but no true cell borders exist Lower RULED_LINE_THRESHOLD sensitivity or catch the empty list and re-route to pdfplumber; the router’s default already favors pdfplumber when in doubt.
pdfplumber extract_table() returns None Page is borderless and whitespace is too irregular for the default table settings Pass tuned table_settings (e.g. {"vertical_strategy": "text"}), or if the page has no text layer, route to the OCR fallback with Tesseract.
GhostscriptNotFound when Camelot runs Ghostscript missing or Camelot installed without the [cv] extra Install camelot-py[cv] and system ghostscript; pdfplumber needs neither, so a Ghostscript-free host should route grid pages carefully.
Two engines disagree on row count One split or merged a spanning cell the other did not Run compare_engines, flag for manual review, and reconcile merged cells per reconciling merged cells in funder budget workbooks.
KeyError reading accuracy_score from a pdfplumber record Treating pdfplumber as if it reports a confidence score pdfplumber records accuracy_score = None; validate its output by column count against the expected schema instead of a confidence number.