Port morss reader options and widen feed discovery

Merged
AS
7h ago
+585-50

This PR was created by an AI agent (OpenHands) on behalf of the user.

This closes the gaps found when checking whether morss and rss-finder can be deleted: pol already had the main reader features, but these pieces were still missing.

What changed

Discovery now tries the page, then the site root

find_feeds used to try one fixed list of absolute paths on the origin. It now tries the well-known paths relative to the page first, then repeats them on the origin when the subpath has no feed. The list also gains atom.json, rss.json, index.json and rss/ from feedfinder-ts, while keeping the blogspot paths pol already had.

flowchart TD
    A[Page HTML] --> B[link rel=alternate and feed-looking anchors]
    A --> C[well-known paths under the page]
    B --> D{verify by parsing}
    C --> D
    D -->|no well-known hit| E[well-known paths under the origin]
    E --> D
    D --> F[feed list, deduplicated by final URL]

morss options

OptionBehaviour
--post STRINGsends the body to the feed URL instead of a GET
--order first|last|newest|oldestpicks which items are enriched when --max-items is reached; output order is unchanged
--firstlinktakes the item link from the first <a> in the description
--nolinkdrops anchors from item content but keeps their text
--norefdrops item links
--silentsuppresses the output

The item options (order, firstlink, nolink, noref) are also accepted as /read query parameters.

Link recovery and item fixes

  • Unwrap translate.google.*, news.google.com/news/url, getpocket.com/redirect and any google.<tld>/url (not only google.com).
  • Decode feedsportal /<token>/story01.htm links.
  • Prefer feedburner:origLink over the wrapper <link> when parsing RSS.
  • Normalise titles longer than 20 characters that are entirely uppercase.
  • Recover the Wikipedia daily-highlight link and the Reddit [link] target, as morss's ItemFix did.

The item link is now repaired for every item before full-text fetching, so --firstlink, Wikipedia and Reddit links point at the real article instead of the wrapper.

One cache fix

Fetcher keys its cache by URL, so a POST response could be read back by a later GET. Non-GET requests now skip the cache read and write.

Tests

tests/test_reader_*.py: 225 passed, 3 skipped (Django and Redis are not installed locally). New coverage includes:

  • page-relative before root candidates, root fallback, JSON well-known paths
  • POST body, silent output, noref, nolink, firstlink, order newest
  • first/last/newest/oldest selection, uppercase titles, Wikipedia and Reddit links
  • Google News/Translate, Pocket and feedsportal links, feedburner origLink
  • POST responses not entering the GET cache

Scope

This does not add feedfinder-ts's GitHub, Reddit and YouTube URL matchers, and does not change the saved-feed preview URL. Those were not part of this request and can be a follow-up if you want full rss-finder parity.

@ashcoft can click here to continue refining the PR


This change is Reviewable


Summary by cubic

Ports the remaining morss reader options and widens feed discovery, closing the gaps that kept morss and rss-finder from being replaced by pol.

Feed discovery

  • Well-known feed paths are tried under the page first, then repeated at the site root as a fallback.
  • The candidate list gains atom.json, rss.json, index.json, and rss/, keeping the blogspot paths that already existed.

morss options and link fixes

  • Adds --post, --order, --firstlink, --nolink, --noref, and --silent; the item options also work as /read query parameters. --order picks which items get enriched when --max-items is hit without reordering the output.
  • Unwraps Google News, Google Translate, any google.<tld>/url, and Pocket redirects, and decodes feedsportal links.
  • RSS parsing prefers feedburner:origLink, and item links are now repaired before full-text fetching so Wikipedia, Reddit, and --firstlink point at the real article.
  • Titles longer than 20 characters that are entirely uppercase are normalized.
  • POST responses no longer read or fill the URL-keyed HTTP cache.

RetriggerView in GreptileConfidence Score: 3/5

Fix the skipped root feeds and malformed-link crash before merging.

Fix All in CodexFindings

  1. P1 Root feeds are skipped ▶
  2. P1 One link crashes the feed ▶
  3. P2 Summary links remain ▶
Fix with agent prompt
### Issue 1
pol/reader/pipeline.py:144
`process()` can no longer find a root-only `/feed.xml` from a page such as `/blog/post`. The 15 page-relative guesses consume most of `candidates[:20]`, leaving only five root guesses. `/feed.xml` is the sixth root guess, so it is never fetched, even when the page advertises no feeds. Users get “No feed found” for a feed the previous code found. Give the root pass its own budget and add a test for this case.

### Issue 2
pol/reader/urls.py:169-172
`decode_feedsportal()` raises `IndexError` when a matching token ends in `0` or contains `00`: splitting on `0` produces an empty string, then `pair[0]` fails. For example, an item URL ending in `/abcdefghijklmnopqrs0/story01.htm` crashes the whole feed. This runs on every item and does not check the host. Reject malformed tokens and keep the original URL instead.

```suggestion
    pairs = match.group(1).split('0')[1:]
    if not pairs or any(not pair for pair in pairs):
        return url
    decoded = ''.join(
        _FEEDSPORTAL_CODES.get(pair[0], pair[0]) + pair[1:]
        for pair in pairs
    )
```

### Issue 3
pol/reader/pipeline.py:258-259
`--nolink` leaves anchors in `item.summary`. Summary-only RSS items therefore still contain links, including in JSON's `content_html`, because `Item.body()` falls back to the summary. `--clip` also adds the untouched summary back above full text. Strip anchors from both `item.summary` and `item.content`.

```suggestion
        if options.nolink:
            item.summary = strip_links(item.summary)
            item.content = strip_links(item.content)
```

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR adds morss-style reader options, repairs more item links, and tries page-relative feed paths before root paths.

  • Feed discovery checks paths under the page before the site root.
  • Reader options choose which items get enriched and what links they show.
  • Feed items recover article links from common wrappers and feed formats.
  • The CLI can POST a body or finish quietly without printing a feed.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Read URL] --> B{Feed parses?}
    B -->|No| C[Advertised feeds]
    C --> D[15 page-relative guesses]
    D --> E[Root guesses]
    E --> F[Shared limit can skip root feeds]
    B -->|Yes| G[Repair item links]
    G --> H[Select items to enrich]
    H --> I[Apply item options]
    I --> J[Format output]

Reviews (1) · Last reviewed commit: "feat(reader): port morss options and wid..." · Reviewed by Greptile

Summary by CodeRabbit

  • New Features

    • Added options to submit requests with a POST body, control item ordering and limits, and choose how links and references are handled.
    • Added a silent mode that completes successfully without writing results to standard output or a file.
    • Expanded feed discovery to check paths relative to the page and site root, including JSON feeds.
    • Improved handling of article links, redirects, and item titles.
  • Bug Fixes

    • Prevented POST responses from being reused as cached GET responses.
    • Improved selection of original links in feed items.
  • Documentation

    • Updated the README with the new options and feed-discovery behavior.