Yjx"dZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZejeZehdZdZdd eDZejd ezd zejZejd ezd zejZd8dZehdZehdZd9dZd:dZ d;dZ!dd#Z$ d?d@d$Z%dAd%Z&dBd(Z'ehd)Z(dCd+Z)d?dDd.Z*dEd/Z+ d?dFd6Z,gd7Z-dS)GuRouting helpers for inbound user-attached images. Two modes: native — attach images as OpenAI-style ``image_url`` content parts on the user turn. Provider adapters (Anthropic, Gemini, Bedrock, Codex, OpenAI chat.completions) already translate these into their vendor-specific multimodal formats. text — run ``vision_analyze`` on each image up-front and prepend the description to the user's text. The model never sees the pixels; it only sees a lossy text summary. This is the pre-existing behaviour and still the right choice for non-vision models. The decision is made once per message turn by :func:`decide_image_input_mode`. It reads ``agent.image_input_mode`` from config.yaml (``auto`` | ``native`` | ``text``, default ``auto``) and the active model's capability metadata. In ``auto`` mode: - If the active model reports ``supports_vision=True`` (via config override or models.dev metadata), we attach natively — vision-capable main models should always see the original pixels, even when an auxiliary vision backend is configured. That auxiliary backend then acts as a *fallback* for sessions whose main model can't take images. - Otherwise, if the user has explicitly configured ``auxiliary.vision`` (provider/model/base_url not ``auto``/empty), we route through the text pipeline so the auxiliary vision backend can describe the image for the text-only main model. - Otherwise (non-vision model, no explicit override), we fall back to text via the default vision_analyze flow. This keeps ``vision_analyze`` surfaced as a tool in every session — skills and agent flows that chain it (browser screenshots, deeper inspection of URL-referenced images, style-gating loops) keep working. The routing only affects *how user-attached images on the current turn* are presented to the main model. ) annotationsN)Path)AnyDictListOptionalTuple>autotextnative) .png.jpg.jpeg.gif.webp.bmpz.tiffz.tifz.heic|c#@K|]}|dVdS).N)lstrip).0es HC:\Users\server\AppData\Local\hermes\hermes-agent\agent\image_routing.py r?s,AAahhsmmAAAAAAz/(?\"']+?\.(?:z)(?:\?[^\s<>\"']*)?r strreturnTuple[List[str], List[str]]c t|tr|sggfSg tjd|tjD]=} ||f>tjd|D]=} ||f>d fd }g}t}t|D]}||r | d}tj |} tj |sun#t$rYwxYw||vr||||g}t} t"|D]y}||r | d} | d } | | vrO| | || z||fS) uScan free-form text for image references the model should see. Returns ``(local_paths, urls)``: * ``local_paths`` — absolute (``/``) or home-relative (``~/``) paths whose suffix is an image extension AND whose expanded form exists on disk as a file. Order-preserving, deduplicated. * ``urls`` — ``http(s)://…`` URLs whose path ends in an image extension (a ``?query`` is allowed after the extension). Order-preserving, deduplicated. Matches inside fenced code blocks (``` ``` ```) and inline backticks (`` `…` ``) are skipped so that snippets pasted into a task body for reference aren't mistaken for live attachments. This mirrors the behaviour of ``gateway.platforms.base.BaseAdapter.extract_local_files``. Local paths are validated against the filesystem; URLs are not (the provider fetches them at request time). z```[^\n]*\n.*?```z `[^`\n]+`posintrboolc<tfdDS)Nc3>K|]\}}|cxko|kncVdSN)rsrr s rrz7extract_image_refs.._in_code..rs;77DAq1<<<._in_codeqs'7777J777777rrz .,;:!?)]>)r r!rr") isinstancerrefinditerDOTALLappendstartendset_LOCAL_IMAGE_PATH_REgroupospath expanduserisfileOSErroradd _IMAGE_URL_RErstrip) r mr* local_paths seen_pathsmatchrawexpandedurls seen_urlsurlr)s @rextract_image_refsrFRss( dC 2v )+J [-tRY ? ?0017799aeegg.//// [t , ,0017799aeegg.////888888 K55J%..t44%% 8EKKMM " "  kk!nn7%%c** 7>>(++      H  z ! ! x   8$$$$D%%I''--   8EKKMM " "  kk!nnjj%% )    c C  sE66 FF>1onyestrue>0noofffalserArOptional[bool]c*t|tr|St|tr|dvrt|SdSt|tr<|}|t vrdS|tvrdSdS)z@Return True/False for recognised boolean values, None otherwise.)rNTF)r+r"r!rstriplower _TRUE_TOKENS _FALSE_TOKENS)rAr's r_coerce_capability_boolrVs#t #s &==99 t#s IIKK      4  5 4rcfgOptional[Dict[str, Any]]providermodelct|tsdS|d}t|tr|ni}t|d}||St |dpd}d}|dr|tdd}|d}t|tr|ni} ttd|||fD]} | | } t| tr| ni} | d} t| tr| ni}||}t|tr|ni}t|d}||cS|d }t|trjt}td||fD]o} | | | dr+| | tddW| d| p|D]} t| tst | d pd}||vrS| d} t| tr| ni}||}t|tr|ni}t|d}||cSdS) uResolve user-declared vision capability from config.yaml. Resolution order, first hit wins: 1. ``model.supports_vision`` (top-level shortcut for the active model) 2. ``providers..models..supports_vision`` (named custom providers — ``provider`` may be the runtime-resolved value ``"custom"`` and/or the user-declared name under ``model.provider``; both are tried. For ``custom:`` syntax, the stripped ```` is also tried as a provider key.) Returns None when no override is set, so the caller falls through to models.dev. Returns False explicitly only when the user wrote a recognised boolean false token. NrZsupports_visionrYcustom: providersmodelscustom_providersname) r+dictgetrVrrR startswithlenfromkeysfilterlistr2r:)rWrYrZ model_cfg_raw model_cfgtopconfig_providerstripped_suffix providers_raw providers_cfgp entry_rawentry models_raw models_cfg per_model_raw per_modelcoercedracandidate_names entry_names r_supports_vision_overrider{sh& c4 tGGG$$M1;M41P1P X VXI !)--0A"B"B C CC  )-- 339r::@@BBOO!!),,;)#i..//:GGK((M5? t5T5T$\MMZ\M ]]6$?O(TUU V V  !%%a(( -7 4-H-H P bYYx(( 3=j$3O3O%WZZUW "u-- 5? t5T5T$\MMZ\ ))--8I*J*JKK  NNN  ww122"D)) #uux9:: 3 3A    " " "||I&& 3##Ac)nnoo$67777##MaMM2222)  Ii.. Y]]6228b99??AAJ00"x00J'1*d'C'CKJ&NN511M)3M4)H)HP bI-immz._resolve_inference_base_url..3s 66617799666rrb)agent.auxiliary_clientr}rrR Exceptionr+rcrdr2rhr:rSresplitri)rWrYr}runtimerjrkr~rmryrqrprbrsburaloweredrrrzs r_resolve_inference_base_urlrs1  AAAAAA,23399;;  N        c4 rGGG$$M1;M41P1P X VXI9==,,23399;;H)-- 339r::@@BBO #O D8_5 6 6//A 7799   * * /   Q 2 3 3 3 3    ! . . . .GGK((M-&&#  D!%%d++E%&& :..4"55;;==IIIww122"D)) 66o666)  Ii.. Y]]6228b99??AAJ00Z5E5E5G5Gw5V5VY]]:..4"55;;==B    2s ,0 ==r~r"c|pd}|dkrdS|sdS ddlm}||dkS#t$rYdSwxYw)zBTrue when the active provider likely fronts a local Ollama server.r]ollamaTFr)detect_local_server_type)rRrSagent.model_metadatarr)rYr~rqrs r_should_probe_ollama_visionrAs R  &&((AH}}t uAAAAAA''11X== uusA AAct|tsdS|}|tvr|SdS)z5Normalize a config value into one of the valid modes.r )r+rrRrS _VALID_MODES)rAvals r _coerce_moderPsG c3  v ))++    C l 6rcht|tsdS|dpi}t|tsdS|dpi}t|tsdSt|dpd}t|dpd}t|dpd}|dvr|s|sdSd S) ucTrue when the user configured a specific auxiliary vision backend. An explicit override means the user has a dedicated vision backend available; it's used as a *fallback* when the main model can't take images natively. In ``auto`` mode, native vision on a vision-capable main model still wins over this fallback — see issue #29135. F auxiliaryvisionrYr]rZr~>r]r T)r+rcrdrrRrS)rWauxrrYrZr~s r_explicit_aux_vision_overriderZs  c4 u ''+   $"C c4 u WWX   $"F fd # #u6::j))/R006688>>@@H  7##)r * * 0 0 2 2E6::j))/R006688H<hu 4rc4t|||}||S|r|sdSd} ddlm}|||}n4#t$r'}td|||Yd}~nd}~wwxYw|t |jSt||}|s.|pd dkrd}t||rL ddl m }|||} | | Sn4#t$r'}td |||Yd}~nd}~wwxYwdS) aReturn True/False if we can resolve caps, None if unknown. Consults the user's ``supports_vision`` override in config.yaml first (so custom/local models declared as vision-capable don't fall through to text routing in ``auto`` mode), then falls back to models.dev. Nr)get_model_capabilitiesu2image_routing: caps lookup failed for %s:%s — %sr]rzhttp://localhost:11434/v1)query_ollama_supports_visionu:image_routing: ollama vision probe failed for %s:%s — %s)r{agent.models_devrrloggerdebugr"r\rrRrSrrr) rYrZrWoverridecapsrexcr~r ollama_visions r_lookup_supports_visionrus)h>>H 5t Da;;;;;;%%h66 aaa I8UZ\_````````a D()))*399H /R..006688HDD."8X66   I I I I I I88IIM($$)    LLL           4s,2 A#AA# C$$ D.DDc@d}t|trN|dpi}t|tr"t|d}|dkrdS|dkrdSt |||}|durdSt |rdSdS)a1Return ``"native"`` or ``"text"`` for the given turn. Args: provider: active inference provider ID (e.g. ``"anthropic"``, ``"openrouter"``). model: active model slug as it would be sent to the provider. cfg: loaded config.yaml dict, or None. When None, behaves as auto. r agentimage_input_moder r T)r+rcrdrrr)rYrZrWmode_cfg agent_cfgsupportss rdecide_image_input_modersH#tGGGG$$* i & & G#IMM2D$E$EFFH8x6v 'x<GIF87aGIF89a image/gif sRIFFsWEBP image/webpsBM image/bmpsftyp>avifavisz image/avif>heicheimheisheixhevchevxmif1msf1z image/heic>II*MM*z image/tiffsz image/x-iconisrrrrOptional[bytes]ct ddlm}n+#t$rtdYdSwxYw ddl}|n#t$rYnwxYw ddl}n#t$rYnwxYw ddl m }| ||5}|j dvr| d}|}||dd |cdddS#1swxYwYdS#t$r&}td |Yd}~dSd}~wwxYw) aDecode arbitrary image bytes with Pillow and re-encode as PNG. Returns None if Pillow isn't installed or can't decode the input (rare formats, corrupted bytes, missing optional decoder plugin for HEIC/AVIF, or vector formats like SVG). Caller falls back to skipping the image so the rest of the turn still works. HEIC/HEIF and AVIF need optional Pillow plugins; we try to register them on demand and swallow ImportError so a missing plugin just looks like 'Pillow can't decode this' rather than crashing. r)Imagezimage_routing: Pillow not installed; cannot transcode non-standard image format to PNG. Install with `pip install Pillow` (and `pillow-heif` / `pillow-avif-plugin` for those formats).N)BytesIO>LPLARGBRGBArPNGF)formatoptimizez> " " " " " " " " " " " " " " " " " "  JC   ttttt sl $11A AAA$$ A10A15$DAC:- D:C>>DC>D D7D22D7r6rc|t|}|r|Stjt|\}}|r|dr|S|j}ddddddd|dS) zReturn image MIME type for *path*. If *raw* bytes are provided, magic-byte sniffing wins (authoritative). Otherwise we fall back to ``mimetypes`` then suffix-based defaults. Nzimage/rrrrr)rrr rrr)r mimetypes guess_typerresuffixrSrd)r6rAsniffedmime_rs r _guess_mimerKs  (--  N"3t99--GD! )) [   F    c&, rc ddlm}|t|n?#t$r'}td||Yd}~dSd}~wt $rYnwxYw |}n4#t $r'}td||Yd}~dSd}~wwxYwt||}|tvrTt|}|td||dSt d|j ||}d }tj|d }d |d |S) uWEncode a local image as a base64 data URL at its native size. Size limits are NOT enforced here — the agent retry loop (``run_agent._try_shrink_image_parts_in_messages``) shrinks on the provider's first rejection. Keeping this simple means providers that accept large images (OpenAI 49 MB+, Gemini 100 MB) don't pay a silent quality tax just because one other provider is stricter. Format compatibility IS handled here: if the sniffed MIME isn't one of ``_UNIVERSALLY_SUPPORTED_MIMES`` (i.e. it's something like AVIF, HEIC, BMP, TIFF, or ICO that some providers reject outright), we transcode to PNG with Pillow before declaring media_type. This fixes the user-visible "Could not process image" HTTP 400 from Anthropic on Discord-attached AVIF/HEIC/BMP files. Returns None if the file can't be read OR if the format isn't universally supported AND Pillow can't transcode it (Pillow missing, HEIC/AVIF plugin missing, vector format like SVG, corrupt bytes). The caller reports those paths in ``skipped`` and the rest of the turn proceeds. r)raise_if_read_blockedz6image_routing: blocked local image attachment %s -- %sNu'image_routing: failed to read %s — %s)rAzimage_routing: %s is %s which is not accepted by all major vision providers and could not be transcoded to PNG; skipping this attachment.zIimage_routing: transcoded %s (%s) -> image/png for provider compatibilityrasciizdata:z;base64,)agent.file_safetyrr ValueErrorrwarningr read_bytesr_UNIVERSALLY_SUPPORTED_MIMESrrrbbase64 b64encodedecode)r6rrrAr transcodedb64s r_file_to_data_urlres,  ;;;;;;c$ii(((( OQUWZ[[[ttttt      oo @$LLLttttt t % % %D ///&s++   NN,d     4 W It     3   & &w / /C &4 & & & &&s2! AA  AA!A66 B'B""B' user_text image_paths List[str] image_urlsOptional[List[str]]&Tuple[List[Dict[str, Any]], List[str]]cg}g}g}g}|D]}t|}|r|s#|t |\t |} | s#|t ||dd| id|t ||pgD]J} | pd} | s|dd| id|| K|pd} |s|r~| pd} g} | d|D| d|D| dd | z}d |d g}||||fSg}| r|d | d ||fS) uBuild an OpenAI-style ``content`` list for a user turn. Shape: [{"type": "text", "text": "...\n\n[Image attached at: /local/path]"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}, {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}}, ...] Local paths are read from disk and embedded as base64 ``data:`` URLs. Remote URLs (``http(s)://``) are passed through verbatim — the provider fetches them server-side. The model still sees the pixels either way. For each successfully attached image, a hint is appended to the text part: * local path → ``[Image attached at: ]`` * URL → ``[Image attached: ]`` The hint gives the model a string handle so MCP/skill tools that take an image path or URL argument can be invoked on the same image without an extra round-trip. This parallels the text-mode hint produced by ``Runner._enrich_message_with_vision`` (``vision_analyze using image_url: ``) so behaviour is consistent across both image input modes. Images are attached at their native size. If a provider rejects the request because an image is too large (e.g. Anthropic's 5 MB per-image ceiling), the agent's retry loop transparently shrinks and retries once — see ``run_agent._try_shrink_image_parts_in_messages``. Returns (content_parts, skipped). Skipped entries are local paths that couldn't be read from disk; URLs are never skipped (they're not validated here). image_urlrE)typerr]zWhat do you see in this image?c3"K|] }d|dV dS)z[Image attached at: ]Nr&)rrqs rrz-build_native_content_parts..s.NN!5555NNNNNNrc3"K|] }d|dV dS)z[Image attached: rNr&)rus rrz-build_native_content_parts..s.JJq2a222JJJJJJrz  r )rr ) rexistsis_filer/rrrRextendjoin)rrrskipped image_partsattached_paths attached_urlsraw_pathrqdata_urlrEr base_text hint_lines combined_textpartss rbuild_native_content_partsrs\LG(*K "N!M - - NNxxzz   NN3x== ) ) ) $Q''  NN3x== ) ) ) *      c(mm,,,,R""yb!!        S!!!! O " " $ $D<< " NN~NNNNNNJJMJJJJJJ$***TYYz-B-BB 06 'N'N&O [!!!g~ E 5 fd33444 '>r)rrrF)r rrr)rArrrO)rWrXrYrrZrrrO)rWrXrYrrr)rYrr~rrr")rArrr)rWrXrr"r%)rYrrZrrWrXrrO)rYrrZrrWrXrr)rArrr)rArrr)r6rrArrr)r6rrr)rrrrrrrr).__doc__ __future__rrloggingrr5r,pathlibrtypingrrrrr getLogger__name__r frozensetr _IMAGE_EXTSr _IMAGE_EXT_PATTERNcompile IGNORECASEr3r;rFrTrUrVr{rrrrrrrrrrrr__all__r&rrr"s$$L#"""""  33333333333333  8 $ $y33344  XXAA[AAAAA "rz69KKfTM !$669OOM BBBBZy33344  55566 "QQQQh6666r    <%)*****Z    d0000| )y***   1111h     48'8'8'8'|'+VVVVVr   r