{
  "skill": "alterlab-arxiv",
  "evals": [
    {
      "id": "keyword-category-search",
      "prompt": "Find me recent preprints on sparse autoencoders and mechanistic interpretability in machine learning — I want the latest stuff in cs.LG, maybe the 20 newest.",
      "expected_output": "Triggers the arXiv skill. Runs a keyword search (e.g. arxiv_search.py with --keywords 'sparse autoencoders' 'mechanistic interpretability') constrained to the cs.LG category, sorted by submittedDate, returning structured JSON with titles, abstracts, authors, categories, and links. Does not claim citation counts (arXiv has none).",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "cs.LG" },
        { "type": "behavior", "value": "Performs a keyword search filtered by arXiv category sorted by submission date." }
      ]
    },
    {
      "id": "arxiv-id-lookup",
      "prompt": "Pull the metadata and PDF link for arXiv 2309.10668 and 2406.04093 so I can add them to my reading list.",
      "expected_output": "Triggers the arXiv skill. Resolves the bare arXiv IDs via ID lookup (--ids / get_by_ids), returning title, authors, abstract, categories, published/updated dates, DOI, and pdf_url for each. Treats the values as arXiv identifiers, not search keywords.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "2309.10668" },
        { "type": "behavior", "value": "Resolves specific arXiv IDs to their metadata and PDF links rather than running a free-text search." }
      ]
    },
    {
      "id": "author-tracking",
      "prompt": "Track the recent cs.LG preprints coming out of Anthropic — I want to monitor what their interpretability team is posting.",
      "expected_output": "Triggers the arXiv skill. Explains that the arXiv API's au: prefix matches author names only (au:anthropic returns nothing; affiliations are not searchable), so it builds an au:\"...\" OR query over the team's known author names, combined with cat:cs.LG and abstract terms like interpretability, sorted by submittedDate descending, and returns the recent preprint list. Uses arXiv query syntax with uppercase boolean operators and may point to alterlab-openalex for institution-level tracking.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "behavior", "value": "Constructs an author-name arXiv query (au: prefix) for the group's members rather than an affiliation search, and flags that arXiv cannot filter by affiliation." }
      ]
    },
    {
      "id": "near-miss-pubmed",
      "prompt": "I need peer-reviewed clinical journal articles on metformin and cardiovascular outcomes — indexed, with MeSH terms.",
      "expected_output": "Should NOT trigger the arXiv skill. This is biomedical, peer-reviewed, MeSH-indexed literature, which arXiv does not cover (arXiv is preprints in physics/math/CS/quant-bio). The skill should defer to the pubmed database skill (alterlab-pubmed) for indexed clinical journal articles.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "pubmed" }
      ]
    },
    {
      "id": "near-miss-openalex",
      "prompt": "For the 'Attention Is All You Need' transformer paper, how many times has it been cited and what's its citation trajectory over the years?",
      "expected_output": "Should NOT trigger the arXiv skill. arXiv provides no citation counts or impact metrics. The request for citation counts and a citation trajectory should defer to the openalex database skill (alterlab-openalex), which exposes citation and impact data.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "openalex" }
      ]
    }
  ]
}

# arXiv API Reference

## Overview

The arXiv API provides programmatic access to preprint metadata via an Atom XML feed. It supports search queries with field-specific operators, boolean logic, ID-based retrieval, sorting, and pagination. No authentication required.

## Base URL

```
https://export.arxiv.org/api/query
```

(The docs historically show `http://`; the endpoint also serves HTTPS, which the script uses.)

**Backend migration (2025-11-11).** arXiv moved this API to a new cloud implementation
with the same URL and parameters. Visible differences: malformed queries now return
**HTTP 400** with an Atom error entry instead of an empty 200; bare terms are
auto-prefixed (`pineapple` → `all:pineapple`) unless you pass `raw=1`; abstracts are
no longer hard-wrapped at 80 characters.

## Rate Limiting

- Terms of use: **no more than one request every three seconds, one connection at a
  time**, counted across all machines you control (https://info.arxiv.org/help/api/tou.html)
- Aggressive crawling will result in temporary IP bans
- Use `time.sleep(3)` between requests
- Include a descriptive `User-Agent` header

## Query Parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `search_query` | Query string with field prefixes and boolean operators | (none) |
| `id_list` | Comma-separated arXiv IDs | (none) |
| `start` | Starting index for pagination (0-based) | `0` |
| `max_results` | Number of results to return (max 2000 per call) | `10` |
| `sortBy` | Sort field: `relevance`, `lastUpdatedDate`, `submittedDate` | `relevance` |
| `sortOrder` | Sort direction: `ascending`, `descending` | `descending` |

**Note**: `search_query` and `id_list` can be used together (results are ANDed) or separately.

## Search Query Syntax

### Field Prefixes

| Prefix | Field | Example |
|--------|-------|---------|
| `ti:` | Title | `ti:transformer` |
| `au:` | Author name (not affiliation) | `au:bengio` |
| `abs:` | Abstract | `abs:attention mechanism` |
| `co:` | Comment | `co:accepted at NeurIPS` |
| `jr:` | Journal Reference | `jr:Nature` |
| `cat:` | Category | `cat:cs.LG` |
| `all:` | All fields | `all:deep learning` |
| `id:` | arXiv ID | `id:2309.10668` |

### Boolean Operators

Operators **must** be uppercase:

```
ti:transformer AND abs:attention           # Both conditions
au:bengio OR au:lecun                      # Either condition
cat:cs.LG ANDNOT cat:cs.CV                # Exclude category
```

### Grouping

Use parentheses for complex queries:

```
(ti:sparse AND ti:autoencoder) AND cat:cs.LG
au:"Chris Olah" AND (abs:interpretability OR abs:alignment)
(cat:cs.LG OR cat:cs.CL) AND ti:reinforcement learning
```

### Phrase Search

Quotes for exact phrases:

```
ti:"sparse autoencoder"
au:"Yoshua Bengio"
abs:"reinforcement learning from human feedback"
```

### Date ranges

`submittedDate` (and `lastUpdatedDate`) accept an inclusive range in GMT,
`[YYYYMMDDTTTT TO YYYYMMDDTTTT]`:

```
cat:cs.LG AND submittedDate:[202501010000 TO 202501312359]
```

### Affiliations

There is no affiliation field to search. `au:anthropic` returns nothing; `au:openai`
only matches papers that list the organization as a collective author. Query a group
through its members' names, or use OpenAlex's institution filter.

### Wildcards

Not supported by the arXiv API. Use broader terms and filter client-side.

## Example Requests

### Basic keyword search
```
GET http://export.arxiv.org/api/query?search_query=all:sparse+autoencoder&max_results=10
```

### Author + category
```
GET http://export.arxiv.org/api/query?search_query=au:bengio+AND+cat:cs.LG&max_results=50&sortBy=submittedDate
```

### ID lookup
```
GET http://export.arxiv.org/api/query?id_list=2309.10668,2406.04093
```

### Combined search + ID
```
GET http://export.arxiv.org/api/query?search_query=cat:cs.LG&id_list=2309.10668
```

### Paginated results
```
# Page 1 (results 0-99)
GET ...?search_query=cat:cs.LG&start=0&max_results=100&sortBy=submittedDate

# Page 2 (results 100-199)
GET ...?search_query=cat:cs.LG&start=100&max_results=100&sortBy=submittedDate
```

## Response Format (Atom XML)

The API returns an Atom 1.0 XML feed.

### Feed-level elements

```xml
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:arxiv="http://arxiv.org/schemas/atom">

  <title>ArXiv Query: ...</title>
  <id>http://arxiv.org/api/...</id>
  <updated>2024-01-15T00:00:00-05:00</updated>

  <!-- Total results available (not just returned) -->
  <opensearch:totalResults>1500</opensearch:totalResults>
  <opensearch:startIndex>0</opensearch:startIndex>
  <opensearch:itemsPerPage>50</opensearch:itemsPerPage>

  <entry>...</entry>
  <entry>...</entry>
</feed>
```

### Entry elements

```xml
<!-- Illustrative entry: structure only; values are placeholders. -->
<entry>
  <!-- Unique identifier (includes version) -->
  <id>http://arxiv.org/abs/YYMM.NNNNNv2</id>

  <!-- Dates -->
  <published>2023-09-19T17:58:00Z</published>
  <updated>2023-10-04T14:22:00Z</updated>

  <!-- Metadata -->
  <title>Example Paper Title...</title>
  <summary>Full abstract text...</summary>

  <!-- Authors -->
  <author>
    <name>First Author</name>
  </author>
  <author>
    <name>Second Author</name>
  </author>

  <!-- Categories -->
  <arxiv:primary_category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
  <category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
  <category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>

  <!-- Links -->
  <link href="http://arxiv.org/abs/YYMM.NNNNNv2" rel="alternate" type="text/html"/>
  <link href="http://arxiv.org/pdf/YYMM.NNNNNv2" rel="related" type="application/pdf" title="pdf"/>

  <!-- Optional (present only when the author/arXiv supplies them) -->
  <arxiv:comment>42 pages, 30 figures</arxiv:comment>
  <arxiv:doi>10.1103/PhysRevLett.XX.XXXXXX</arxiv:doi>
  <arxiv:journal_ref>...</arxiv:journal_ref>
</entry>
```

### Entry field descriptions

| Field | Description |
|-------|-------------|
| `id` | Canonical arXiv URL with version (e.g., `http://arxiv.org/abs/2309.10668v2`) |
| `published` | First submission date (ISO 8601) |
| `updated` | Last update date (ISO 8601) |
| `title` | Paper title (may contain line breaks in XML) |
| `summary` | Full abstract text |
| `author/name` | Author full name (one per `<author>` element) |
| `arxiv:primary_category` | Primary arXiv category |
| `category` | All categories (multiple elements) |
| `link[@type='text/html']` | Abstract page URL |
| `link[@title='pdf']` | PDF download URL |
| `arxiv:comment` | Author comment (page count, conference, etc.) |
| `arxiv:doi` | Associated DOI (if exists) |
| `arxiv:journal_ref` | Journal publication reference (if published) |

## Complete Category List

### Computer Science (cs.*)

| Category | Name |
|----------|------|
| `cs.AI` | Artificial Intelligence |
| `cs.AR` | Hardware Architecture |
| `cs.CC` | Computational Complexity |
| `cs.CE` | Computational Engineering, Finance, and Science |
| `cs.CG` | Computational Geometry |
| `cs.CL` | Computation and Language |
| `cs.CR` | Cryptography and Security |
| `cs.CV` | Computer Vision and Pattern Recognition |
| `cs.CY` | Computers and Society |
| `cs.DB` | Databases |
| `cs.DC` | Distributed, Parallel, and Cluster Computing |
| `cs.DL` | Digital Libraries |
| `cs.DM` | Discrete Mathematics |
| `cs.DS` | Data Structures and Algorithms |
| `cs.ET` | Emerging Technologies |
| `cs.FL` | Formal Languages and Automata Theory |
| `cs.GL` | General Literature |
| `cs.GR` | Graphics |
| `cs.GT` | Computer Science and Game Theory |
| `cs.HC` | Human-Computer Interaction |
| `cs.IR` | Information Retrieval |
| `cs.IT` | Information Theory |
| `cs.LG` | Machine Learning |
| `cs.LO` | Logic in Computer Science |
| `cs.MA` | Multiagent Systems |
| `cs.MM` | Multimedia |
| `cs.MS` | Mathematical Software |
| `cs.NA` | Numerical Analysis |
| `cs.NE` | Neural and Evolutionary Computing |
| `cs.NI` | Networking and Internet Architecture |
| `cs.OH` | Other Computer Science |
| `cs.OS` | Operating Systems |
| `cs.PF` | Performance |
| `cs.PL` | Programming Languages |
| `cs.RO` | Robotics |
| `cs.SC` | Symbolic Computation |
| `cs.SD` | Sound |
| `cs.SE` | Software Engineering |
| `cs.SI` | Social and Information Networks |
| `cs.SY` | Systems and Control |

### Statistics (stat.*)

| Category | Name |
|----------|------|
| `stat.AP` | Applications |
| `stat.CO` | Computation |
| `stat.ME` | Methodology |
| `stat.ML` | Machine Learning |
| `stat.OT` | Other Statistics |
| `stat.TH` | Statistics Theory |

### Mathematics (math.*)

| Category | Name |
|----------|------|
| `math.AC` | Commutative Algebra |
| `math.AG` | Algebraic Geometry |
| `math.AP` | Analysis of PDEs |
| `math.AT` | Algebraic Topology |
| `math.CA` | Classical Analysis and ODEs |
| `math.CO` | Combinatorics |
| `math.CT` | Category Theory |
| `math.CV` | Complex Variables |
| `math.DG` | Differential Geometry |
| `math.DS` | Dynamical Systems |
| `math.FA` | Functional Analysis |
| `math.GM` | General Mathematics |
| `math.GN` | General Topology |
| `math.GR` | Group Theory |
| `math.GT` | Geometric Topology |
| `math.HO` | History and Overview |
| `math.IT` | Information Theory |
| `math.KT` | K-Theory and Homology |
| `math.LO` | Logic |
| `math.MG` | Metric Geometry |
| `math.MP` | Mathematical Physics |
| `math.NA` | Numerical Analysis |
| `math.NT` | Number Theory |
| `math.OA` | Operator Algebras |
| `math.OC` | Optimization and Control |
| `math.PR` | Probability |
| `math.QA` | Quantum Algebra |
| `math.RA` | Rings and Algebras |
| `math.RT` | Representation Theory |
| `math.SG` | Symplectic Geometry |
| `math.SP` | Spectral Theory |
| `math.ST` | Statistics Theory |

### Physics

| Category | Name |
|----------|------|
| `astro-ph` | Astrophysics (+ subcategories: .CO, .EP, .GA, .HE, .IM, .SR) |
| `cond-mat` | Condensed Matter (+ subcategories) |
| `gr-qc` | General Relativity and Quantum Cosmology |
| `hep-ex` | High Energy Physics - Experiment |
| `hep-lat` | High Energy Physics - Lattice |
| `hep-ph` | High Energy Physics - Phenomenology |
| `hep-th` | High Energy Physics - Theory |
| `math-ph` | Mathematical Physics |
| `nlin` | Nonlinear Sciences (+ subcategories) |
| `nucl-ex` | Nuclear Experiment |
| `nucl-th` | Nuclear Theory |
| `physics` | Physics (+ subcategories: .comp-ph, .data-an, .bio-ph, etc.) |
| `quant-ph` | Quantum Physics |

### Quantitative Biology (q-bio.*)

| Category | Name |
|----------|------|
| `q-bio.BM` | Biomolecules |
| `q-bio.CB` | Cell Behavior |
| `q-bio.GN` | Genomics |
| `q-bio.MN` | Molecular Networks |
| `q-bio.NC` | Neurons and Cognition |
| `q-bio.OT` | Other Quantitative Biology |
| `q-bio.PE` | Populations and Evolution |
| `q-bio.QM` | Quantitative Methods |
| `q-bio.SC` | Subcellular Processes |
| `q-bio.TO` | Tissues and Organs |

### Quantitative Finance (q-fin.*)

| Category | Name |
|----------|------|
| `q-fin.CP` | Computational Finance |
| `q-fin.EC` | Economics |
| `q-fin.GN` | General Finance |
| `q-fin.MF` | Mathematical Finance |
| `q-fin.PM` | Portfolio Management |
| `q-fin.PR` | Pricing of Securities |
| `q-fin.RM` | Risk Management |
| `q-fin.ST` | Statistical Finance |
| `q-fin.TR` | Trading and Market Microstructure |

### Electrical Engineering and Systems Science (eess.*)

| Category | Name |
|----------|------|
| `eess.AS` | Audio and Speech Processing |
| `eess.IV` | Image and Video Processing |
| `eess.SP` | Signal Processing |
| `eess.SY` | Systems and Control |

### Economics (econ.*)

| Category | Name |
|----------|------|
| `econ.EM` | Econometrics |
| `econ.GN` | General Economics |
| `econ.TH` | Theoretical Economics |

## Pagination

The API returns at most 2000 results per request, and at most 30000 results in total across paginated calls (requests beyond that return HTTP 400). For larger result sets, paginate in slices of <=2000:

```python
all_results = []
start = 0
batch_size = 100

while True:
    params = {
        "search_query": "cat:cs.LG",
        "start": start,
        "max_results": batch_size,
        "sortBy": "submittedDate",
        "sortOrder": "descending",
    }
    results = fetch(params)  # your fetch function
    if not results:
        break
    all_results.extend(results)
    start += batch_size
    time.sleep(3)  # respect rate limit
```

The total number of results available is in the `opensearch:totalResults` element of the feed.

## Downloading Papers

### PDF
```
http://arxiv.org/pdf/{arxiv_id}
http://arxiv.org/pdf/{arxiv_id}v{version}
```

### Abstract page
```
http://arxiv.org/abs/{arxiv_id}
```

### Source (LaTeX)
```
http://arxiv.org/e-print/{arxiv_id}
```

### HTML (experimental)
```
http://arxiv.org/html/{arxiv_id}
```

## arXiv ID Formats

| Format | Era | Example |
|--------|-----|---------|
| `YYMM.NNNNN` | 2015+ | `2309.10668` |
| `YYMM.NNNN` | 2007-2014 | `0706.0001` |
| `archive/YYMMNNN` | Pre-2007 | `hep-th/9901001` |

All formats are accepted by the API.

## Common Pitfalls

1. **Boolean operators must be UPPERCASE**: `AND`, `OR`, `ANDNOT` (lowercase is treated as search terms)
2. **URL encoding**: Spaces in queries must be encoded as `+` or `%20`
3. **No full-text search**: The API only searches metadata (title, abstract, authors, etc.)
4. **Errors vs. empty results**: zero hits return a feed with `opensearch:totalResults` = 0 and no entries. Invalid queries or IDs return HTTP 400 with a single entry whose id starts with `https://arxiv.org/api/errors` and whose `<summary>` explains the problem (e.g. `Invalid query string: '(ti:sparse'`) — report it, don't treat it as "no papers"
5. **Version numbering**: `published` date is v1 submission; `updated` is latest version date
6. **Rate limiting**: Exceeding limits can result in 403 errors or temporary bans
7. **Max 2000 per request**: `max_results` is capped at 2000 per call; retrieve more via `start`-based pagination (30000 total cap)

## External Resources

- arXiv API documentation: https://info.arxiv.org/help/api/index.html
- arXiv API user manual: https://info.arxiv.org/help/api/user-manual.html
- arXiv bulk data access: https://info.arxiv.org/help/bulk_data.html
- arXiv category taxonomy: https://arxiv.org/category_taxonomy
- OAI-PMH interface (for bulk metadata): https://oaipmh.arxiv.org/oai (the old `export.arxiv.org/oai2` URL now 301-redirects here)
- API change announcements: https://groups.google.com/a/arxiv.org/g/api

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.9"
# dependencies = ["requests>=2.31"]
# ///
"""
arXiv Search Tool
Search and retrieve preprints from arXiv via the Atom API.
Supports keyword search, author search, category filtering, ID lookup, and PDF download.

Run with uv (resolves the requests dependency automatically):
    uv run scripts/arxiv_search.py --keywords "diffusion models" --category cs.LG
"""

import requests
import json
import argparse
import xml.etree.ElementTree as ET
import time
import sys
import os
import re
from typing import List, Dict, Optional


class ArxivSearcher:
    """Search interface for arXiv preprints via the Atom API."""

    BASE_URL = "https://export.arxiv.org/api/query"
    ATOM_NS = "{http://www.w3.org/2005/Atom}"
    ARXIV_NS = "{http://arxiv.org/schemas/atom}"

    VALID_SORT_BY = ["relevance", "lastUpdatedDate", "submittedDate"]
    VALID_SORT_ORDER = ["ascending", "descending"]
    VALID_SEARCH_FIELDS = ["ti", "au", "abs", "co", "jr", "cat", "all", "id"]

    def __init__(self, verbose: bool = False, delay: float = 3.0):
        self.verbose = verbose
        self.delay = delay
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": "ArxivSearchTool/1.0 (scientific-skills)"
        })
        self._last_request_time = 0.0

    def _log(self, message: str):
        if self.verbose:
            print(f"[INFO] {message}", file=sys.stderr)

    def _rate_limit(self):
        """Enforce minimum delay between requests."""
        elapsed = time.time() - self._last_request_time
        if elapsed < self.delay:
            wait = self.delay - elapsed
            self._log(f"Rate limiting: waiting {wait:.1f}s")
            time.sleep(wait)
        self._last_request_time = time.time()

    def _parse_entry(self, entry: ET.Element) -> Dict:
        """Parse a single Atom entry into a dict."""
        def text(tag, ns=None):
            ns = ns or self.ATOM_NS
            el = entry.find(f"{ns}{tag}")
            return el.text.strip() if el is not None and el.text else ""

        # Authors
        authors = []
        for author_el in entry.findall(f"{self.ATOM_NS}author"):
            name_el = author_el.find(f"{self.ATOM_NS}name")
            if name_el is not None and name_el.text:
                authors.append(name_el.text.strip())

        # Categories
        categories = []
        primary_category = ""
        for cat_el in entry.findall(f"{self.ATOM_NS}category"):
            term = cat_el.get("term", "")
            if term:
                categories.append(term)
        prim_el = entry.find(f"{self.ARXIV_NS}primary_category")
        if prim_el is not None:
            primary_category = prim_el.get("term", "")

        # Links
        pdf_url = ""
        abs_url = ""
        for link_el in entry.findall(f"{self.ATOM_NS}link"):
            href = link_el.get("href", "")
            link_type = link_el.get("type", "")
            link_title = link_el.get("title", "")
            if link_title == "pdf" or link_type == "application/pdf":
                pdf_url = href
            elif link_type == "text/html" or (not link_type and "/abs/" in href):
                abs_url = href

        # Extract arXiv ID from the Atom id field
        raw_id = text("id")
        arxiv_id = re.sub(r"^https?://arxiv\.org/abs/", "", raw_id)
        # Strip version suffix for the canonical ID
        arxiv_id_bare = re.sub(r"v\d+$", "", arxiv_id)

        return {
            "arxiv_id": arxiv_id_bare,
            "title": " ".join(text("title").split()),  # collapse whitespace
            "authors": authors,
            "abstract": " ".join(text("summary").split()),
            "categories": categories,
            "primary_category": primary_category,
            "published": text("published"),
            "updated": text("updated"),
            "doi": text("doi", self.ARXIV_NS),
            "comment": text("comment", self.ARXIV_NS),
            "journal_ref": text("journal_ref", self.ARXIV_NS),
            "pdf_url": pdf_url,
            "abs_url": abs_url or f"https://arxiv.org/abs/{arxiv_id}",
        }

    def _error_summary(self, response: requests.Response) -> Optional[str]:
        """Return the message of an arXiv API error entry, if the feed holds one."""
        try:
            root = ET.fromstring(response.text)
        except ET.ParseError:
            return None
        for entry in root.findall(f"{self.ATOM_NS}entry"):
            id_el = entry.find(f"{self.ATOM_NS}id")
            if id_el is not None and id_el.text and "/api/errors" in id_el.text:
                summary = entry.find(f"{self.ATOM_NS}summary")
                return (summary.text or "").strip() if summary is not None else id_el.text
        return None

    def _fetch(self, params: Dict) -> List[Dict]:
        """Execute API request and parse results.

        Raises ValueError when arXiv rejects the query: since the Nov 2025 backend
        migration, malformed queries return HTTP 400 with an Atom error entry
        (previously an empty 200), and that must not be reported as "0 results".
        """
        self._rate_limit()
        self._log(f"Query params: {params}")

        response = self.session.get(self.BASE_URL, params=params, timeout=30)
        error = self._error_summary(response)
        if error:
            raise ValueError(f"arXiv rejected the query: {error}")
        response.raise_for_status()

        root = ET.fromstring(response.text)
        entries = root.findall(f"{self.ATOM_NS}entry")
        self._log(f"Parsed {len(entries)} entries")

        results = []
        for entry in entries:
            parsed = self._parse_entry(entry)
            # Skip any empty placeholder entry
            if not parsed["title"] or parsed["arxiv_id"] == "":
                continue
            results.append(parsed)

        return results

    def search(
        self,
        query: str,
        max_results: int = 50,
        start: int = 0,
        sort_by: str = "relevance",
        sort_order: str = "descending",
    ) -> List[Dict]:
        """
        Search arXiv with a query string.

        Args:
            query: arXiv query string (e.g., "ti:transformer AND cat:cs.LG")
            max_results: Maximum number of results (max 2000 per request;
                paginate with `start` for larger sets, up to 30000 total)
            start: Starting index for pagination
            sort_by: One of "relevance", "lastUpdatedDate", "submittedDate"
            sort_order: "ascending" or "descending"

        Returns:
            List of paper dicts
        """
        if sort_by not in self.VALID_SORT_BY:
            raise ValueError(f"sort_by must be one of {self.VALID_SORT_BY}")
        if sort_order not in self.VALID_SORT_ORDER:
            raise ValueError(f"sort_order must be one of {self.VALID_SORT_ORDER}")

        # arXiv caps a single call at 2000 results (slices of at most 2000).
        max_results = min(max_results, 2000)

        params = {
            "search_query": query,
            "start": start,
            "max_results": max_results,
            "sortBy": sort_by,
            "sortOrder": sort_order,
        }

        return self._fetch(params)

    def get_by_ids(self, arxiv_ids: List[str]) -> List[Dict]:
        """
        Retrieve papers by their arXiv IDs.

        Args:
            arxiv_ids: List of arXiv IDs (e.g., ["2309.10668", "2406.04093"])

        Returns:
            List of paper dicts
        """
        # Clean IDs: strip URLs, versions
        clean_ids = []
        for aid in arxiv_ids:
            aid = re.sub(r"^https?://arxiv\.org/abs/", "", aid.strip())
            aid = re.sub(r"v\d+$", "", aid)
            clean_ids.append(aid)

        params = {
            "id_list": ",".join(clean_ids),
            "max_results": len(clean_ids),
        }

        return self._fetch(params)

    def download_pdf(self, arxiv_id: str, output_path: str) -> bool:
        """
        Download a paper's PDF.

        Args:
            arxiv_id: arXiv ID (e.g., "2309.10668")
            output_path: File path or directory to save to

        Returns:
            True if successful
        """
        arxiv_id = re.sub(r"^https?://arxiv\.org/abs/", "", arxiv_id.strip())
        arxiv_id = re.sub(r"v\d+$", "", arxiv_id)

        pdf_url = f"https://arxiv.org/pdf/{arxiv_id}"
        self._log(f"Downloading: {pdf_url}")

        # If output_path is a directory, generate filename
        if os.path.isdir(output_path):
            filename = arxiv_id.replace("/", "_") + ".pdf"
            output_path = os.path.join(output_path, filename)

        self._rate_limit()

        try:
            response = self.session.get(pdf_url, timeout=60)
            response.raise_for_status()

            os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
            with open(output_path, "wb") as f:
                f.write(response.content)

            self._log(f"Saved to: {output_path}")
            return True
        except Exception as e:
            self._log(f"Download error: {e}")
            return False

    @staticmethod
    def build_query(
        title: Optional[str] = None,
        author: Optional[str] = None,
        abstract: Optional[str] = None,
        category: Optional[str] = None,
        all_fields: Optional[str] = None,
    ) -> str:
        """
        Build an arXiv query string from components.

        Args:
            title: Search in title
            author: Search by author name
            abstract: Search in abstract
            category: Filter by category (e.g., "cs.LG")
            all_fields: Search all fields

        Returns:
            arXiv query string
        """
        parts = []
        if all_fields:
            parts.append(f"all:{all_fields}")
        if title:
            parts.append(f"ti:{title}")
        if author:
            parts.append(f"au:{author}")
        if abstract:
            parts.append(f"abs:{abstract}")
        if category:
            parts.append(f"cat:{category}")

        return " AND ".join(parts)


def main():
    parser = argparse.ArgumentParser(
        description="Search arXiv preprints",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  %(prog)s --keywords "sparse autoencoder" --category cs.LG --max-results 20
  %(prog)s --author "Yoshua Bengio" --max-results 50
  %(prog)s --ids 2309.10668 2406.04093
  %(prog)s --query "ti:GRPO AND cat:cs.LG" --sort-by submittedDate
  %(prog)s --ids 2309.10668 --download-pdf papers/
        """,
    )

    parser.add_argument("--verbose", "-v", action="store_true")

    search_group = parser.add_argument_group("Search options")
    search_group.add_argument("--keywords", "-k", nargs="+", help="Keywords to search")
    search_group.add_argument("--author", "-a", help="Author name")
    search_group.add_argument("--ids", nargs="+", help="arXiv IDs to look up")
    search_group.add_argument("--query", "-q", help="Raw arXiv query string")
    search_group.add_argument(
        "--search-field",
        choices=ArxivSearcher.VALID_SEARCH_FIELDS,
        default="all",
        help="Field to search keywords in (default: all)",
    )

    filter_group = parser.add_argument_group("Filter options")
    filter_group.add_argument("--category", "-c", help="arXiv category (e.g., cs.LG)")
    filter_group.add_argument("--max-results", type=int, default=50, help="Max results (default: 50, max: 2000 per call)")
    filter_group.add_argument(
        "--sort-by",
        choices=ArxivSearcher.VALID_SORT_BY,
        default=None,
        help="Sort field (default: relevance; submittedDate when browsing a category)",
    )
    filter_group.add_argument(
        "--sort-order",
        choices=ArxivSearcher.VALID_SORT_ORDER,
        default="descending",
    )

    output_group = parser.add_argument_group("Output options")
    output_group.add_argument("--output", "-o", help="Output JSON file (default: stdout)")
    output_group.add_argument("--download-pdf", help="Download PDFs to this directory")

    args = parser.parse_args()
    searcher = ArxivSearcher(verbose=args.verbose)

    # --- ID lookup ---
    if args.ids:
        if args.download_pdf:
            for aid in args.ids:
                searcher.download_pdf(aid, args.download_pdf)
            return 0

        results = searcher.get_by_ids(args.ids)
        query_desc = f"id_list:{','.join(args.ids)}"

    # --- Raw query ---
    elif args.query:
        query = args.query
        if args.category and f"cat:{args.category}" not in query:
            query = f"({query}) AND cat:{args.category}"

        results = searcher.search(
            query=query,
            max_results=args.max_results,
            sort_by=args.sort_by or "relevance",
            sort_order=args.sort_order,
        )
        query_desc = query

    # --- Keyword search ---
    elif args.keywords:
        field = args.search_field
        keyword_parts = [f'{field}:"{kw}"' if " " in kw else f"{field}:{kw}" for kw in args.keywords]
        query = " AND ".join(keyword_parts)
        if args.category:
            query = f"({query}) AND cat:{args.category}"

        results = searcher.search(
            query=query,
            max_results=args.max_results,
            sort_by=args.sort_by or "relevance",
            sort_order=args.sort_order,
        )
        query_desc = query

    # --- Author search ---
    elif args.author:
        query = f'au:"{args.author}"'
        if args.category:
            query = f"{query} AND cat:{args.category}"

        results = searcher.search(
            query=query,
            max_results=args.max_results,
            sort_by=args.sort_by or "relevance",
            sort_order=args.sort_order,
        )
        query_desc = query

    # --- Category browse ---
    elif args.category:
        query = f"cat:{args.category}"
        results = searcher.search(
            query=query,
            max_results=args.max_results,
            sort_by=args.sort_by or "submittedDate",
            sort_order=args.sort_order,
        )
        query_desc = query

    else:
        parser.error("Provide --keywords, --author, --ids, --query, or --category")
        return 1

    # Output
    output_data = {
        "query": query_desc,
        "result_count": len(results),
        "results": results,
    }

    output_json = json.dumps(output_data, indent=2, ensure_ascii=False)

    if args.output:
        os.makedirs(os.path.dirname(args.output) or ".", exist_ok=True)
        with open(args.output, "w") as f:
            f.write(output_json)
        print(f"Results written to {args.output}", file=sys.stderr)
    else:
        print(output_json)

    return 0


if __name__ == "__main__":
    try:
        sys.exit(main())
    except (ValueError, requests.exceptions.RequestException) as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(2)