This PR was created by an AI agent (OpenHands) on behalf of the user.
This closes the gaps found when checking whether morss and rss-finder can be deleted: pol already had the main reader features, but these pieces were still missing.
What changed
Discovery now tries the page, then the site root
find_feeds used to try one fixed list of absolute paths on the origin. It now tries the well-known paths relative to the page first, then repeats them on the origin when the subpath has no feed. The list also gains atom.json, rss.json, index.json and rss/ from feedfinder-ts, while keeping the blogspot paths pol already had.
flowchart TD
A[Page HTML] --> B[link rel=alternate and feed-looking anchors]
A --> C[well-known paths under the page]
B --> D{verify by parsing}
C --> D
D -->|no well-known hit| E[well-known paths under the origin]
E --> D
D --> F[feed list, deduplicated by final URL]
morss options
Option
Behaviour
--post STRING
sends the body to the feed URL instead of a GET
--order first|last|newest|oldest
picks which items are enriched when --max-items is reached; output order is unchanged
--firstlink
takes the item link from the first <a> in the description
--nolink
drops anchors from item content but keeps their text
--noref
drops item links
--silent
suppresses the output
The item options (order, firstlink, nolink, noref) are also accepted as /read query parameters.
Link recovery and item fixes
Unwrap translate.google.*, news.google.com/news/url, getpocket.com/redirect and any google.<tld>/url (not only google.com).
Decode feedsportal /<token>/story01.htm links.
Prefer feedburner:origLink over the wrapper <link> when parsing RSS.
Normalise titles longer than 20 characters that are entirely uppercase.
Recover the Wikipedia daily-highlight link and the Reddit [link] target, as morss's ItemFix did.
The item link is now repaired for every item before full-text fetching, so --firstlink, Wikipedia and Reddit links point at the real article instead of the wrapper.
One cache fix
Fetcher keys its cache by URL, so a POST response could be read back by a later GET. Non-GET requests now skip the cache read and write.
Tests
tests/test_reader_*.py: 225 passed, 3 skipped (Django and Redis are not installed locally). New coverage includes:
page-relative before root candidates, root fallback, JSON well-known paths
POST body, silent output, noref, nolink, firstlink, order newest
first/last/newest/oldest selection, uppercase titles, Wikipedia and Reddit links
Google News/Translate, Pocket and feedsportal links, feedburner origLink
POST responses not entering the GET cache
Scope
This does not add feedfinder-ts's GitHub, Reddit and YouTube URL matchers, and does not change the saved-feed preview URL. Those were not part of this request and can be a follow-up if you want full rss-finder parity.
Ports the remaining morss reader options and widens feed discovery, closing the gaps that kept morss and rss-finder from being replaced by pol.
Feed discovery
Well-known feed paths are tried under the page first, then repeated at the site root as a fallback.
The candidate list gains atom.json, rss.json, index.json, and rss/, keeping the blogspot paths that already existed.
morss options and link fixes
Adds --post, --order, --firstlink, --nolink, --noref, and --silent; the item options also work as /read query parameters. --order picks which items get enriched when --max-items is hit without reordering the output.
Unwraps Google News, Google Translate, any google.<tld>/url, and Pocket redirects, and decodes feedsportal links.
RSS parsing prefers feedburner:origLink, and item links are now repaired before full-text fetching so Wikipedia, Reddit, and --firstlink point at the real article.
Titles longer than 20 characters that are entirely uppercase are normalized.
POST responses no longer read or fill the URL-keyed HTTP cache.
Confidence Score: 3/5
Fix the skipped root feeds and malformed-link crash before merging.
### Issue 1
pol/reader/pipeline.py:144
`process()` can no longer find a root-only `/feed.xml` from a page such as `/blog/post`. The 15 page-relative guesses consume most of `candidates[:20]`, leaving only five root guesses. `/feed.xml` is the sixth root guess, so it is never fetched, even when the page advertises no feeds. Users get “No feed found” for a feed the previous code found. Give the root pass its own budget and add a test for this case.
### Issue 2
pol/reader/urls.py:169-172
`decode_feedsportal()` raises `IndexError` when a matching token ends in `0` or contains `00`: splitting on `0` produces an empty string, then `pair[0]` fails. For example, an item URL ending in `/abcdefghijklmnopqrs0/story01.htm` crashes the whole feed. This runs on every item and does not check the host. Reject malformed tokens and keep the original URL instead.
```suggestion
pairs = match.group(1).split('0')[1:]
if not pairs or any(not pair for pair in pairs):
return url
decoded = ''.join(
_FEEDSPORTAL_CODES.get(pair[0], pair[0]) + pair[1:]
for pair in pairs
)
```
### Issue 3
pol/reader/pipeline.py:258-259
`--nolink` leaves anchors in `item.summary`. Summary-only RSS items therefore still contain links, including in JSON's `content_html`, because `Item.body()` falls back to the summary. `--clip` also adds the untouched summary back above full text. Strip anchors from both `item.summary` and `item.content`.
```suggestion
if options.nolink:
item.summary = strip_links(item.summary)
item.content = strip_links(item.content)
```
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Summary
The PR adds morss-style reader options, repairs more item links, and tries page-relative feed paths before root paths.
Feed discovery checks paths under the page before the site root.
Reader options choose which items get enriched and what links they show.
Feed items recover article links from common wrappers and feed formats.
The CLI can POST a body or finish quietly without printing a feed.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Read URL] --> B{Feed parses?}
B -->|No| C[Advertised feeds]
C --> D[15 page-relative guesses]
D --> E[Root guesses]
E --> F[Shared limit can skip root feeds]
B -->|Yes| G[Repair item links]
G --> H[Select items to enrich]
H --> I[Apply item options]
I --> J[Format output]