Kzzzza 中

How do agents manage a research knowledge base?

2026 words 5 min #Agent

This page is machine-translated and may contain errors. Read the Chinese original

Recently I've spent quite a bit of time figuring out how to get AI agents to help me read papers more effectively. The process went through roughly three stages, each one fixing the problems of the previous stage. Looking back, it's actually pretty interesting, so I'm writing it down.

Stage 1: Dragging papers into a web dialog box

The initial idea was very simple — throw the paper PDF at an AI and let it read it for me. This was also the most common AI-assisted way of reading papers before local agents became widespread.

I tried basically all of them — Zotero's built-in AI-assisted reading, various web-based paper Q&A tools. The workflow was simple: upload PDF → AI parses it → I ask questions → it answers.

However, there are some obvious problems:

  1. The context window is not enough: a normal CS paper converted to plain text is roughly 8,000–15,000 words. Add formulas, tables, and references, and it easily exceeds the context limit of many models. Even if you manage to squeeze it in, after a few rounds of dialogue the earlier content gets pushed out. As you keep asking, the AI starts losing its memory — one moment you're discussing the experiments section, and suddenly it forgets which dataset the paper used. Moreover, sometimes you need to compare papers directly or write a literature review, which requires the AI to understand more papers, making the context window limitation even more severe.

  2. Also, it has to analyze from scratch every time: I might read the same paper several times, understanding it from different angles. But every new conversation is a fresh start — a paper I had the AI read carefully yesterday has to be re-uploaded and re-parsed today just to ask a different question.

  3. Writing paper notes: sometimes the need isn't just reading the paper, but also creating reading notes based on it. I like to include some of the paper's main figures in my notes, but that's impossible with a web-only AI.

  4. Not just papers: in the broad CS field I mainly follow, papers almost always have corresponding open-source code repositories. The true nature of a method often lies in the code — hyperparameter tuning tricks, engineering optimizations, implementation details that the paper glosses over. But in the toolchain at the time, papers were papers and code was code, with no intersection at all. After reading a paper, I still had to go to GitHub myself, dig through the code, and build the connection between the code and the paper.

Stage 2: Hand-Rolling a Paper Survey Skill

After I started using Claude Code, I picked up some ways to use Skills, so a while back I built a paper-notes assistant skill.

Combining the Agent capabilities of Claude Code, I wrote a skill specifically for producing paper surveys. In simple terms, what it does is: based on the papers I provide (paper title, PDF, URL, paper keywords), it handles paper retrieval, download, and analysis, and finally produces a structured paper survey note.

The whole workflow is divided into six stages:

The Complete Seven-Stage Pipeline

  1. Paper Retrieval: Search for and download relevant paper PDFs based on keywords
  2. PDF Parsing: Paper PDFs are converted to Markdown via the MinerU API, with formulas, images, tables, and layout structure fully extracted
  3. Deep Single-Paper Analysis: Each paper launches an independent subagent that reads the full text closely in an isolated context window, extracting the research problem, core contributions, methodology, experimental results, and limitations according to a template, generating a Markdown analysis document for each paper
  4. Cross-Paper Correlation: All single-paper analyses are aggregated to find thematic clusters, methodological lineages, formula/symbol correspondences, consensus and conflicts, and research gaps, generating a Markdown document of paper correlations and overview
  5. Outline Design: Based on the correlation analysis, produce a structured blog outline, specifying what each section covers and which papers to cite
  6. Draft Writing: Write the main text section by section according to the outline
  7. Final Polish: Check citation consistency, LaTeX correctness, paragraph transitions, hook and conclusion quality

In practice, the quality of the surveys it produces is genuinely good. Stage 3, cross-paper correlation, is especially impressive — the agent can find connections I might not notice myself. For example, several papers that look completely unrelated in direction actually correspond to each other in their underlying mathematical frameworks.

But there are still some unresolved problems:

  1. Knowledge is completely non-reusable: Every time I write a new survey, the relevant papers have to be downloaded, parsed, and analyzed from scratch. Papers that were already deeply analyzed in a previous survey have to be redone entirely just because the topic changed. MinerU's free quota is limited, and parsing long papers takes anywhere from tens of seconds to minutes — re-parsing is pure waste.

  2. Code is still missing: The Skill only processes paper PDFs and doesn't touch code repositories at all. Algorithm details, hyperparameter tuning lore, and engineering implementation knowledge — I still have to dig through GitHub myself. The Agent is completely useless when it comes to code.

  3. The pipeline is too heavy: Running all seven stages end-to-end is suitable for the scenario of "I want to write a proper survey." But everyday needs — "Hey, what does this paper say?" "How is it different from the one I read yesterday?" "How is the code written?" — don't need such a heavy process at all, and the paper index content produced by the analysis doesn't leave convenient interfaces for other skills to use.

Phase 3: MAPCE — A Personal RAG Knowledge Base for Papers + Code

So I started wondering: could I take a different approach?

Instead of feeding papers to the Agent every time, why not first build my own RAG knowledge base and let the Agent look things up on its own when needed?

The idea is actually pretty simple. Reading papers, bookmarking papers, tagging them in Zotero — all of that is essentially accumulating a personal knowledge base. If that accumulated material could be directly accessed by the Agent, it wouldn't have to start from scratch every time.

So I built MAPCE — MCP-based Agent for Paper and Code Exploration (naming projects is way too hard; every name I pick is already taken...). The goal:

Build a local personal research literature & paper database

How It Works

MAPCE is essentially a local vector database (LanceDB) + an MCP Server.

The indexing side looks roughly like this:

1
PDF/arXiv/代码仓库 → MinerU解析 → 多层分块 → 向量嵌入 → LanceDB 存储

Both papers and code are chunked hierarchically, which is the most core design decision in the whole system:

Paper side: three-level chunking + figures and tables

  • L1 Paper-level: title + abstract + keywords (~200–500 tokens) → for coarse filtering
  • L2 Section-level: full content of each section (~500–2000 tokens) → for fine-grained ranking
  • L3 Paragraph-level: individual paragraphs (~200–500 tokens) → for context injection
  • Additional image chunks and table chunks: title + file path / HTML content

Since the embedding model used for RAG is not multimodal, images and tables — which are special and important information — need to be handled separately.

Code side: five-level chunking + configuration files

  • L1 Repo-level: README summary + directory tree + entry points + build info
  • L2 File-level: file path + imports + function/class signature list
  • L3 Function/class-level: full implementation + call relationships calls[] + called-by relationships called_by[]
  • L4 Module-level: top-level constants, registry decorators, __main__ blocks
  • L5 Test-level: test functions + associated symbols under test
  • Configuration file chunks: hyperparameters and configs from YAML/JSON/TOML

The database also stores the correspondence between papers and code repositories.

On the retrieval side, I used a progressive four-stage strategy: first determine the query intent (paper-oriented, code-oriented, or mixed), then L1 coarse filtering to top-20, then L2 fine ranking within the candidate set, and finally pull adjacent paragraphs, call graphs, figures, and tables around the best matches — all within a reasonable token budget.

Finally, 8 tools are exposed to Claude Code via the MCP protocol: search papers, search code, index a new paper, index a code repository, list indexed papers, get paper overview, delete a paper, and view statistics. The Agent uses these tools as naturally as querying a database — I don't need to manually dig through anything.

Data Sources

Currently four sources are supported for adding papers:

  • arXiv: search the arXiv API and batch download for indexing
  • Local PDF: single file or recursive directory scan, with automatic deduplication
  • Zotero: directly read the local Zotero SQLite database — works even while Zotero is open, by automatically copying to a temporary file to bypass the lock (this feature took quite a while to debug)
  • URL: just give an arXiv link and it downloads

I also implemented three-layer deduplication (arxiv_id exact match → DOI exact match → title embedding similarity > 0.95), so the same paper added from different sources won't be indexed twice.

What It Solves, and What's Still Missing

MAPCE seems? to solve some problems:

  • Knowledge reuse. A paper is indexed once and stays in the library forever. No matter what topic I switch to or what direction of survey I'm writing, previously accumulated papers can always be retrieved. MinerU API calls went from "N papers × per survey" to "1 × per paper," saving a lot of quota.

  • Papers and code can be searched together. The same library contains both paper passages and corresponding code implementations. Searching "how is this attention variant written" returns both the formula description from the paper and the attention.py from the repository. In Phase 2, this was completely blank.

  • Everyday queries are lightweight enough. "What is this paper about" or "what do the recently indexed diffusion papers have in common" — just search directly, no need to spin up the six-stage pipeline.

  • Incremental updates. New papers can be added anytime, with automatic deduplication that doesn't affect existing data.

Of course, MAPCE is currently just a rough prototype — a "Unit Zero":

  • Code analysis is fairly crude. Python parses decently with AST, but C++/CUDA can only be handled with regex hacks — template metaprogramming and complex macro expansion are basically unmanageable. Many paper repositories are also multi-language, and Makefiles, Dockerfiles, and shell scripts aren't covered yet.
  • Retrieval quality depends heavily on the chunking strategy. Chunk too coarsely and retrieval is imprecise; chunk too finely and context is insufficient. The current three/five-level scheme is a compromise after several rounds of iteration — fine for most scenarios, but struggles with papers and code repos that have unusual structures.
  • No automatic updates. arXiv papers get new versions, code repos get new commits — currently all require manual re-indexing. The ideal would be subscription-based automatic tracking, preferably running on a personal server with a loop that automatically detects changes and asks the user via WeChat or Feishu whether to update the database.
  • No frontend. Right now it's purely MCP tools and command-line operations. Convenient for the Agent, but not very friendly for users who want to directly browse indexed content.
  • Insufficient testing. The database is currently only at the "it works" level. Whether it can actually provide good knowledge to the agent, whether it saves tokens, and what the retrieval recall rate is — all still need follow-up testing. ~My own estimate is that there's still plenty of room for performance improvement.~
  • Deep integration with Zotero. Right now the project's connection to Zotero is just extracting PDFs from papers. Could we also add the user's Zotero notes to the database, or provide a Zotero-integrated agent plugin?

Finally

Looking back at the three stages, it's really just a process of constantly trying to make myself lazier.

Writing this far, I realize it seems like a process of constantly reinventing the wheel. But each wheel I built did solve some real problems. From "dragging papers into the chat box," to skills, to "building a personal knowledge base" — every step was driven by actual needs, none of it was technology for technology's sake.

There's still plenty I want to do next. If you're interested in these directions or have suggestions, feel free to come discuss, open issues, or submit PRs at MAPCE](https://github.com/kzzzza/mapce).