_resolve_wp_tag_ids created a WordPress tag for every keyword the rewriter
invented, up to 12 per post. That is where the 3.055 tags for 955 posts
came from - 1.683 of them used exactly once, 473 attached to no post at
all. The categories are stable now, but the tags would simply grow back.
Three changes, all on the write path:
A proposed tag has to appear for wordpress_new_tag_min_proposals (3)
different articles before it is created. Proposals are counted in the new
tag_proposals table, keyed on (name, article), so re-publishing an article
does not inflate its own count. Tags that already exist in WordPress are
assigned as before - the gate only guards creation.
Only the first wordpress_max_tags_per_post (5) tags reach WordPress. The
full list still feeds the category rules, which were validated against it.
The lookup fallback of reusing the first search hit is gone. It filed
"Camping" under the unrelated existing tag "Campingplatz" whenever the
exact tag was missing, which quietly produced wrong tags rather than none.
If the proposal bookkeeping fails, nothing is creatable that run: existing
tags still get assigned and the taxonomy stays put, rather than falling
back to creating everything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first pass only covered published posts, so 137 scheduled ones were
still uncategorised. Seven of them matched no rule at all and exposed
real gaps: the regions Pfalz and Erzgebirge, and the gear nouns in
"Duschzelte", "Outdoor-Messer" and "Outdoor-Stuhl", which the existing
compound keywords could not reach because the noun sits at the end.
Adding those flipped two discount posts ("Lidl Zelt im Angebot", "Aldi
Relax-Stuhl") into the gear category, because the new product keywords
outweighed the weaker "lidl"/"aldi" signals. Both retailers only ever
appear in discount posts here, so they now carry enough weight to
trigger the offer override themselves.
Region keywords are weight 2, not 3: at 3 they outranked an explicit
"Campingplatz" in the title and pulled campsite articles into
"Reiseziele". Bare "pfalz" is weight 1 because it double-counts inside
"rheinland-pfalz".
Scheduled posts now match a rule 136 of 137 times. Ten published posts
change category as a result and were updated in WordPress to keep the
archive and the rules in sync.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Published articles carried no category, so WordPress filed all of them
under the catch-all "Allgemein" - 941 of 955 posts by the time this was
noticed. The rules here mirror the one-off backfill of 2026-07-31 and
reproduce its result on all 955 posts exactly.
Matching is done on word boundaries rather than plain substrings, which
otherwise filed a campsite in Klagenfurt under "Recht & Vorschriften" via
the keyword "klage". German compounds opt in explicitly with a trailing
"*". A clear discount signal takes precedence over the product topic,
because product tags otherwise outnumber it.
Unknown slugs are never auto-created: unlike tags, the category set is
curated in WordPress, so a mismatch should surface rather than spawn a
new category. Articles that match no rule stay uncategorised on purpose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two bugs caused multiple articles to land on the same publish slot:
1. main.py: asyncio.create_task() returned immediately, allowing a second
pipeline trigger (N8N + Telegram /run or two N8N calls) to start a
second concurrent run. Added asyncio.Lock (_pipeline_lock) so any
second trigger while the pipeline is running is rejected immediately.
2. scheduler.py: reserve_publish_slot() read the list of occupied slots
and wrote the new slot in two separate DB connections. Concurrent threads
could both see the same "free" slot before either committed its write.
Fixed by wrapping the entire read-find-write cycle in a threading.Lock
(_slot_lock) and a single DB connection, so the slot check and the
slot assignment are atomic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New /admin/article-list: paginated (50/page) table with thumbnail,
title, excerpt (120 chars), status, scheduled date, and WP ID input
- Sticky save bar with live change counter (JS tracks modified inputs,
highlights changed cells in amber, disables save when nothing changed)
- POST /admin/article-list/update: saves only changed WP IDs in one
request; clears stale wp_post_url so WP-Sync repopulates it cleanly
- Filter by status + free-text search (title or article ID)
- Pagination with page/filter state preserved through save redirects
- repositories: add list_articles_page() (offset + search) and
bulk_update_wp_post_ids()
- Dashboard nav: add Artikelliste link
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds sync_db_from_wordpress() that treats WordPress as source of truth:
- future posts: update scheduled_publish_at to WP's actual date
- draft posts: clear scheduled_publish_at (not yet scheduled)
- published posts: mark article as 'published' in DB
- trashed/deleted posts: clear wp_post_id + wp_post_url + slot so article
can be re-processed
Exposed via POST /admin/wp-sync with a sync button on the schedule page.
Run after any manual rescheduling in WordPress to bring DB back in sync.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. Article age filter (ingestion.py + config.py):
- New setting pipeline_max_article_age_days=7 (0 = no limit)
- Skip RSS entries older than N days before expensive extract_article()
- Prevents old articles from Google Alerts re-entering pipeline
2. Image URL pre-validation (ingestion.py):
- HEAD request probe for each primary image candidate during ingestion
- Falls back to next-best candidate if primary returns 4xx
- Network errors treated as OK to avoid false negatives on flaky servers
3. Stale WP draft cleanup (pipeline.py):
- Quality gate rejections now delete any pre-existing WP draft (wp_post_id)
- Prevents orphaned drafts when re-running articles that previously had drafts
4. Schedule overview UI (scheduler.py + admin_ui.py + admin_schedule.html):
- New /admin/schedule page showing calendar grid of all booked slots
- Distinguishes Pipeline-DB slots from WordPress-only slots
- Link added to dashboard navigation
5. Retry for failed articles (admin_ui.py + admin_dashboard.html):
- New POST /admin/articles/{id}/retry endpoint: resets to 'new', releases slot
- '🔄 Wiederholen' button shown in dashboard for all 'close' (error) articles
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- scheduler: use wordpress_app_password (not wordpress_password) so
_fetch_wp_occupied_slots() can actually authenticate against the WP
REST API — previously always returned empty set silently
- pipeline: release reserved publish slot when draft creation fails with
a non-ValueError exception (e.g. WP API error), preventing permanently
blocked slots on failed articles
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add individual Telegram message when an article is rejected by quality
gate (too short raw content or rewritten text), so users see each
rejection in real time instead of only in the bulk summary
- Add quality_gate_rejected counter to PipelineStats and result dict
- Show quality gate rejections separately in pipeline-done summary
(✂️ Qualitätsprüfung: N) distinct from score-based rejections
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stage 1 (before OpenAI rewrite): reject if raw content < pipeline_min_words_raw (default 120)
Stage 2 (after rewrite): reject if rewritten text < pipeline_min_words_rewritten (default 150)
Both stages set status='error' with a descriptive note and skip WP draft creation.
The reserved publish slot is released so it stays available for the next article.
Quality rejections don't abort the pipeline — processing continues with the next article.
New config settings (overridable via .env):
PIPELINE_MIN_WORDS_RAW=120
PIPELINE_MIN_WORDS_REWRITTEN=150
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the scheduler started searching from the last scheduled post date,
skipping all free slots in between (e.g. a free slot on Apr 20 would be ignored
if the last post was on May 18).
Now starts scanning from tomorrow, finding the first available slot regardless
of whether earlier dates have gaps — fills the calendar naturally.
Also extended lookahead from 30 to 60 days.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The scheduler previously only checked the local SQLite DB for occupied slots.
Posts created outside the pipeline (e.g. recovery scripts) were invisible,
causing newly scheduled articles to land on already-taken WP dates.
_fetch_wp_occupied_slots() now queries WP /wp/v2/posts?status=future before
each slot assignment. All scheduling functions accept a wp_occupied set.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
WordPress ignores the date field for draft posts and shows "Sofort veröffentlichen"
instead. Setting status=future causes WP to display and honour the scheduled date,
auto-publishing the post at the given time as intended.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If scheduled_publish_at is not set when _do_rewrite_and_draft runs
(e.g. rewrite_and_update_draft called on a review article), reserve
a slot now so the WP draft always receives a future date.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the credit field only captured a marker prefix (e.g. "Foto:") due to
CSS-class-based extraction picking up only the label element, fall back to
regex-extracting the credit line from the full figcaption caption text.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Derive real source hostname from canonical URL when feed name is generic
(e.g. "Google Alerts"), so the link shows "moin.de" instead of "Google Alerts"
- Use _get_image_meta_for_url() (fuzzy URL matching) for image credit lookup
- Use caption field for Bildnachweis since it already contains embedded credits
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Image metadata keys may have query params (e.g. ?w=1200) that differ from
the selected_url stored in image_review. Fall back to comparing URLs without
query string so the figcaption text is correctly found.
Also simplified _build_image_caption: figcaption text already contains the
credit info, so just use caption directly instead of appending the redundant
credit prefix marker.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Google Alerts wraps matched keywords in <b>...</b> tags.
Strip all HTML tags from the title before storing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Google Alerts feed entries use google.com/url?...&url=<encoded_real_url>&...
tracking links. The extractor was fetching the Google redirect page instead
of the actual article, resulting in empty content and no images.
_resolve_google_redirect() extracts the real URL from the 'url' query
parameter before passing it to extract_article(). Non-Google URLs are
returned unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
source_extraction.py:
- New _extract_image_metadata(): extracts figcaption text + copyright/credit
per image URL using 3 strategies (figure+figcaption, data-* attributes,
adjacent credit spans)
- ExtractedArticle gets new image_metadata field
- extracted_article_to_meta() includes image_metadata in stored JSON
pipeline.py:
- After auto image selection, check if selected_url is set
- Articles without usable image → status "no_image" (excluded with Telegram notice)
- PipelineStats and summary report include no_image counter
db.py:
- Add "no_image" to articles status CHECK constraint
- Migration: recreates articles table with updated constraint on existing DBs
workflow.py / main.py:
- Map no_image as own UI status with rewrite/close transitions
wordpress.py:
- _upload_featured_media() accepts image_caption param, sends to WP media
- _get_image_meta_for_url() / _build_image_caption() helpers
- _build_attribution_block(): separator + attribution paragraph at article end
(original link, author, Bildnachweis/credit)
- _build_post_content() appends attribution block
telegram_bot.py:
- notify_pipeline_done() shows 🖼️ no-image count
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- wordpress.py: catch image download/upload failures and skip image
instead of aborting the entire WP draft update
- pipeline.py: add INFO logs at each step of _do_rewrite_and_draft
to trace OpenAI call, tag generation, and WP API call
- telegram_bot.py: add INFO logs around rewrite execution + exc_info
on error for full traceback in logs
- repositories.py: include scheduled_publish_at in get_article_by_id
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rewrites must not use 'wir haben erforscht/berechnet' since the content
comes from a third-party source. The prompt now passes the source name
and instructs GPT to attribute all claims to the original publisher
(e.g. 'laut PiNCAMP', 'die Auswertung zeigt').
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Articles scoring between warn and auto threshold stayed in "new" status,
causing repeated warning notifications on every /run call. Now they are
set to "review" status after the first warning is sent.
The override callback already resets status to "new" before processing,
so the existing flow works correctly. Also include "review" articles in
/rejected command output so they can be acted on.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The N8N App Release Telegram Trigger had overwritten the webhook
registration, pointing it to N8N instead of the RSS-News backend.
This caused all callback_query events (inline buttons) to be lost,
breaking the override/rewrite/discard buttons.
Changes:
- Re-register webhook to https://news.vanityontour.de/telegram/webhook
with both message and callback_query in allowed_updates
- Add _forward_to_n8n_app_release() to proxy unknown bot commands
(e.g. /release) to the N8N App Release webhook, keeping that
workflow functional without needing its own Telegram Trigger
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reserve the publish slot before creating the WP draft so the
scheduled_publish_at timestamp is available when building the post
payload. WordPress receives the `date` field (e.g. 2026-03-24T09:00:00)
which sets the scheduled publish time on the draft.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Webhook returns 200 immediately, processing runs in background task
→ Telegram no longer retries, eliminates duplicate callbacks and 400 errors
- Consolidate answer_callback_query call to top of handler (before heavy work)
- Add logger.info/error for callback actions to aid debugging
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pipeline runs in background via asyncio. Endpoint returns immediately,
results arrive via Telegram notifications.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
