# Mail-driven Tracker (Inbox → Backend upsert)

A recurring class of "I want events in my inbox to populate my local DB
automatically." The general pattern: a poller script reads envelopes via
[himalaya-v2](../email/himalaya-v2) (or any IMAP CLI), classifies each new
mail, and upserts a row in a `local-http-backend` over HTTP.

The motivating example: a job-application tracker where recruiter
replies should land as status updates without manual editing.

## Components

1. **`local-http-backend`** with a single resource (e.g. `Application`).
2. **Poller script** — Python; runs `himalaya envelope list`, classifies,
   POSTs/PUTs against the backend, persists a state file with seen envelope
   IDs so each mail is processed exactly once.
3. **Cron job** — runs the poller on a schedule (e.g. hourly, Mo–Fr 8–17).
4. **WhatsApp alert** — the cron job's final response is auto-delivered to
   the chat that scheduled it; print a compact alert when `new` ≠ `[]`,
   silent when empty.

## The two pitfalls that always bite

These came up on the FIRST run of the worked example and forced a rewrite.
Bake them in from the start.

### 1. Mail IDs are NOT chronologically ordered

IMAP envelope ordering is by **arrival time at the server**, not by the
`Date:` header. In one real session, the "Eingangsbestätigung" (ID 5641)
arrived AFTER the rejection (ID 5645) even though the rejection's mail
date was later — because the recruiter replied first and the auto-confirmation
landed after. Processing envelopes by their sort order means you may
insert a fresh `open` row, then later try to update its status — but the
status update looks for the existing row, fails, and inserts a duplicate.

**Fix:** maintain an **in-memory cache** during a single poller run, keyed
by `(company_lc, position_lc)` → `db_id`. After every POST/PUT, write the
new id into the cache. Lookups check cache first, then DB.

```python
memory_db: dict[tuple[str, str], int] = {}

def track(company: str, position: str, db_id: int) -> None:
    memory_db[(company.lower(), position.lower())] = db_id

def db_company(company: str) -> dict | None:
    # Prefer open entries, then newest. Cache-first.
    ...
```

This single fix eliminates duplicate entries from same-run batches.

### 2. Bulk newsletters match "Bewerbung" subjects

Job-portal platforms (LinkedIn, Stepstone, Xing) send daily digest mails
that mention "Kandidaten wie Dich" or "Mobile Engineer bei X" — none of
those are real recruiter replies, but their subjects WILL contain keywords
like "Bewerbung" or "Stelle". A naive subject-keyword filter fires on
them and floods your DB with noise.

**Fix:** filter in two stages.

```python
NEWSLETTER_SENDER = ("linkedin", "stepstone", "lisa stein", ...)  # bulk
BEWERB_SUBJECT_HINTS = ("bewerbung", "stelle als", ...)

def match_firm(sender: str, subject: str) -> str | None:
    hay = (sender + " " + subject).lower()
    # 1) Bulk-Newsletter ausschließen
    if any(nl in hay for nl in NEWSLETTER_SENDER):
        return None
    # 2) Subject muss nach Bewerbung klingen
    if not any(h in hay for h in BEWERB_SUBJECT_HINTS):
        return None
    # 3) Konkrete Firma matchen
    for needle, name in FIRMS:
        if needle in hay:
            return name
    return None
```

Maintain both lists (`NEWSLETTER_SENDER` and `FIRMS`) — they are the
explicit allow/deny lists. Newsletters grow faster than firms.

## State file

- **Tracked IDs:** every envelope id the poller has seen (matched or not).
  Don't only store matched ids, or you'll re-scan every hour.
- **Trim:** keep at most ~1500 ids; older envelope ids age out of IMAP
  scrollback anyway.
- **Format:** JSON `{ "seen_env_ids": [...], "last_run": "ISO-8601" }`.

## Subject position extraction

`extract_position(subject)` is its own sub-problem. Common patterns to
cover, in priority order:

1. `Deine Bewerbung – <Position> in <Ort>` (look for `–` or `-` separator)
2. `Bewerbung als <Position>` (after "als")
3. `Re: Bewerbung auf die Stelle <Position>` (after "Stelle")
4. `Stelle als <Position>` (less common)
5. Fallback: `"unbekannt"` — the user can fix this manually later via PUT

Subject lines from one firm are NOT consistent. Plan for fallbacks.

## Body fetch (when classification needs it)

The truncated `envelope list` SUBJECT column is ~90 chars and gets cut
mid-word. For classification, you may need to call
`himalaya message read <id>` to get the full subject AND first ~3KB of body.
That's expensive — only do it for envelopes whose truncated subject is
suspicious (≥ 85 chars), AND when the firm/subject classifier already
accepted the row as "probably relevant." Otherwise the poller makes N²
IMAP fetches per hour.

## Cron: PATH handling

Cron jobs run in a fresh session with no `PATH`. Python scripts that
call `subprocess` against local CLIs (like `himalaya.exe`) need to set
`os.environ["PATH"]` at script startup:

```python
HOME = Path(os.environ.get("USERPROFILE") or os.environ.get("HOME") or "...")
os.environ["PATH"] = (
    str(HOME / ".local" / "bin") + os.pathsep
    + os.environ.get("PATH", "")
)
```

If you skip this, `subprocess([HIMALAYA, ...])` fails with "file not found"
even though the file exists.

## Output shape (for the cron job's final-response delivery)

The cron job's last assistant message is auto-delivered to the chat.
Print a compact JSON the agent can parse; the prompt instructions tell
it to summarize or stay silent:

```json
{
  "scanned": 100,
  "new": [
    {
      "env_id": "5654",
      "firm": "SPIE",
      "status": "rejected",
      "category": "ABSAGE",
      "subject": "Deine Bewerbung bei SPIE",
      "action": "PUT update 1 -> 200",
      "app_id": 1
    }
  ]
}
```

Empty `new` → silent. Non-empty → one WhatsApp line per event.

## Verification

After wiring it up:

1. `python check_bewerbungen.py` once with the state file deleted — confirm
   upserts land as expected (no duplicates, no spam-entries).
2. Run a SECOND time without deleting state — confirm `new` is empty.
3. Delete one matching envelope from IMAP (or temporarily add a fake
   sender to `FIRMS`) and confirm a new POST/UPDATE event fires.

If step 1 produces duplicates, you forgot the in-memory cache. If step 1
includes LinkedIn/Stepstone digests, you forgot the two-stage filter.