Peyash's Log

What's New — Thursday, August 13, 2026

← latest digest ← 2026-08-12 2026-08-14 →

Papers 7

  • AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

    UIUC · Salesforce AI Research ▲66

    A strong "builder" model writes an inference-time harness — deterministic code, task routing, enforced answer formatting — that wraps a weaker target model without touching its parameters. On Theory-of-Mind benchmarks (BigToM, HiToM, MMToM-QA, MuMaToM) average target performance nearly doubles, from 0.49 to 0.91. The gains come from converting unstable reasoning into deterministic code rather than from any expansion of the target's own reasoning, and harness quality tracks builder-model reasoning quality monotonically.

    Top paper of the day, and it lands squarely on harness effects — the eval failure mode where the scaffold, not the model, is what a benchmark actually measures. A near-2x swing from harness alone is a strong argument that unharnessed and harnessed numbers are not comparable quantities.

    #evals #harness-effects #agentic-ai #inference-time-scaling HF ↗

  • OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution

    Affiliations not listed on the paper page ▲64

    A red-teaming benchmark of 10,000+ validated scenarios across 50 domains with 500,000+ tools and skills, built on the premise that agent risk lives in persistent shared state modified across long-horizon workflows rather than in single-turn prompts. A black-box Evolutionary Markov Hypergraph Attack evolves the environment via authorized state transitions and reaches a pooled 85.0% attack success rate, with the margin over instruction-only attacks widening from about 2% on simple tasks to over 17% on complex ones.

    The headline is an eval finding: the agent's specific runtime implementation explains a large share of safety variation beyond the underlying model. Safety leaderboards that rank models are therefore partly ranking harnesses — the same conclusion AI4AI reaches from the opposite direction, published the same day.

    #benchmarks #agentic-ai #red-teaming #ai-safety HF ↗

  • Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill

    Affiliations not listed on the paper page ▲45

    Paper generation implemented as thirteen composable skills inside an existing coding assistant rather than a separate agent platform. It splits model judgment from deterministic checkable operations and fixes experiment plans before results are observed, so claims get revised against measured outcomes. Reports 99.5% citation validity, 96.4% figure editability, and fabrication detection rising from 14% for a single-pass draft to 92% with the full review stack. 11.9M tokens, $8.10 and 3.2 hours per manuscript. Names a failure mode called the Self-Refutation Loop, where repeated experiments keep rejecting the original objective.

    The 14%-to-92% fabrication-detection gap quantifies how much hallucination control in long agentic generation comes from review scaffolding rather than the base model — a third harness result on the same day's page.

    #agentic-ai #ai-coding-tools #llm-as-judge #hallucination HF ↗

  • Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

    Affiliations not listed on the paper page ▲20

    Formulates Narrative Commitment Preservation and ships NCP-Bench: 100 narrative environments derived from movie synopses, each with a structured spec — trajectory, commitments, initial facts — checked automatically turn by turn as a player agent adversarially pushes against a narrator agent. The best model tested (GPT-5.2) survives only 42% of runs after 20 turns; fact-conflict rates run 40-68% across models, and only isolated runs satisfy all achievement commitments within 100 turns.

    A long-horizon consistency benchmark with automatic per-turn verification rather than a judge model rating vibes. The finding that linguistic quality does not predict commitment preservation is a clean argument against fluency-shaped metrics.

    #benchmarks #long-horizon #agentic-ai #evals HF ↗

  • The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

    Shanghai AI Laboratory · Shanghai Jiao Tong University · Shanghai Innovation Institute ▲1

    Casts visual tool-use as a causal graph separating observation-mediated paths from action-induced shortcuts, then intervenes at policy, trajectory and step level. Finds two failure modes — "Calling Without Looking", where returned observations have no causal effect on the answer, and "Looking Without Planning", where observations inform the answer but the calling strategy is incoherent. Aggregate accuracy still goes up, which is the point: the gains are not causally attributable to the visual evidence.

    One upvote, and editorially the most interesting result on the page — an accuracy gain that is real in aggregate and fake in mechanism. Directly relevant to credit attribution in any tool-use eval, and the same skepticism transfers to speech and audio tool pipelines.

    #interpretability #evals #tool-use #multimodal HF ↗

  • ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

    Peking University ▲4

    An Environment Simulator, Attacker Agent and User Simulator synthesize executable, stateful adversarial environments for tool-using agents, automatically discovering viable indirect-prompt-injection points and generating environment-specific payloads. Alignment data generated by the same pipeline improves security without costing benign-task utility.

    A benchmark that is also its own data generator. It stalled at 4 upvotes while OpenART covered the same ground at far larger scale with a sharper runtime-versus-model finding, so it reads better as a second data point than as a standalone.

    #benchmarks #agentic-ai #tool-use #ai-safety HF ↗

  • AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research

    Electronic Arts · Simon Fraser University ▲2

    Coding agents are handed a starter world model across eight game environments and a fixed compute budget, then asked to improve it autonomously. Agents beat their starter in 63 of 64 sessions, and 91% of the winning modifications were research-style changes — architecture, objective, representation — rather than hyperparameter tweaks.

    Reports what kind of change won rather than only whether the score moved, which answers the standing critique that agents on automated-research benchmarks mostly grid-search hyperparameters. The 63-of-64 rate still deserves a skeptical read, since a weak starter makes improvement cheap.

    #benchmarks #world-models #evals #agentic-ai HF ↗

Tech radar 24

  • Hugging Face makes STT optional in its speech-to-speech stack

    Andrés Marafioti

    The stack is really a cascade — speech to VAD to STT to LLM to TTS — and with `--stt none` each completed voice turn now goes straight into an audio-input model (tested with Gemma 4 12B and Inkling), so the model reasons over the audio itself rather than over a transcript.

    A concrete, runnable ablation of transcript-bottleneck versus direct audio conditioning — the central architectural question for speech LLMs right now.

    #speech-llm #audio-language-model #cascaded-vs-end-to-end #open-source LinkedIn ↗

  • Three independent benchmarks converge on the same STT ranking

    Brooke Hopkins

    Weekly voice-AI roundup: Coval had Universal-3.5 Pro lead all 31 models at 3.4% WER on a rebuilt 3,500-sample noisy set, now corroborated by Daily/Pipecat and Hugging Face Open ASR. Also ElevenLabs Dubbing v2, Gradium cross-turn live translation, smallest.ai Lightning v3.1 TTS in Agno, Soniox as Retell's STT provider, Deepgram self-hosting on SageMaker, and StudyFetch cutting inference roughly 10x on NVIDIA Riva plus Parakeet.

    The Coval result itself landed on Aug 10 — the corroboration from two more benchmark houses is what is new. Convergence across independent harnesses is the rare case where a vendor accuracy claim becomes externally checkable. (Disclosure: I work at AssemblyAI.)

    #asr #wer #benchmarks #voice-ai LinkedIn ↗

  • Five confounds in scoring agent trajectories

    DAIR.AI

    New Microsoft Research work makes the agent action policy the measured object rather than the final answer: 8 models, 6 parallel benchmarks, 41 languages, 2.38M rollouts. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, reproducibility caps the achievable gap, and one model asked twice disagrees with itself.

    Eval methodology done properly. Degenerate inputs scoring well, chance agreement and self-inconsistency ceilings all generalize well beyond agent traces.

    #evals #agent-benchmarks #metric-design #multilingual LinkedIn ↗

  • Deepgram ships Flux TTS for live conversation

    Scott Stephenson · Pipecat

    A TTS model built for interruptions, mid-call corrections and fast strings of numbers rather than clean read-aloud text. Now GA and live in Pipecat: it reads the whole conversation rather than the next line, giving adaptive tone, consistent pronunciation and native interruption handling with no SSML markup or style tags, at sub-200ms latency.

    Frames TTS quality around conversational failure modes instead of MOS-style naturalness, which changes what the metric has to capture. The no-SSML claim is the real engineering bet — prosody control moving from markup into model context.

    #tts #voice-agents #real-time #model-release LinkedIn ↗

  • LiveKit takes the opposite prosody bet

    LiveKit

    Expressive mode is live in LiveKit Agents: with one flag the LLM emits inline prosody markup — emotion, pacing, pauses — so a voice agent stops delivering bad news in the same bright tone it uses to upsell.

    Explicit prosody tokens from the LLM versus Flux's implicit context-aware synthesis. Two competing designs shipped in the same week, which makes for an unusually clean comparison.

    #tts #prosody #voice-agents #livekit LinkedIn ↗

  • DeepMind ships sign-language-to-text on-device

    Google DeepMind

    SL2T translates American Sign Language to English on Pixel 11, signing directly into Gboard and Live Transcribe. Described as the most capable sign language translation model to date and optimized for real-world use such as one-handed signing.

    Extends transcription and translation to a visual modality under on-device constraints. One-handed-signing robustness is a real-world eval criterion rather than a benchmark number.

    #sign-language #translation #on-device #model-release LinkedIn ↗

  • Catching voice-agent regressions from production logs

    AssemblyAI

    Take 50 production calls, fan out an analysis fleet, re-transcribe with Universal-3 Pro, summarize through an LLM gateway, and score each against a rubric. The run flagged an entity-capture regression and traced it to min-turn-silence set too low.

    A worked example of rubric grading plus re-transcription as a regression harness, root-caused to a specific endpointing parameter. (Disclosure: I work there.)

    #evals #rubric-grading #asr #regression-testing LinkedIn ↗

  • An explicit ordering of the eval-improvement levers

    Hamel Husain

    Thirteen sessions of AI Product Engineering summarized by cost/benefit: optimize retrieval and context first, then improve the system and harness, and only consider post-training your own model once everything else is exhausted. Material released with Shreya Shankar covering retrieval, post-training, inference and evals.

    An ordering of levers from three years working on this problem — a useful prior against reaching for fine-tuning early.

    #evals #llm-ops #harness #methodology LinkedIn ↗

  • Expert rejects the benchmark choice on a KV-cache paper

    Kuntai Du

    A new cross-model KV cache transfer paper cites his earlier DroidSpeak work as baseline; he argues its evaluation does not support the claims — Llama 3.1 and Qwen 3 on MMLU and GSM8K are too weak a setup for an inference-systems result.

    A domain expert publicly calling a benchmark choice inadequate is exactly the critique loop eval methodology needs and rarely gets in public.

    #kv-cache #evals #benchmarks #inference-optimization LinkedIn ↗

  • Orato ships in-house Hindi/Hinglish ASR and TTS

    Diwakar T.

    Purpose-built for real-time Indian calling workflows — customer support, appointment booking, insurance, sales.

    Code-switched Hindi/Hinglish is a genuine weak spot for general-purpose ASR. The model and whatever eval set backs it are both worth watching.

    #asr #tts #code-switching #multilingual-speech LinkedIn ↗

  • faster-whisper as a self-hosted baseline, tested

    Naman Pandey

    Hands-on notes from testing open-source STT models for a self-hosted voice-to-voice pipeline, using faster-whisper as a practical English and Arabic baseline.

    A practitioner data point on where faster-whisper still holds up as a self-hosted default, and where it does not.

    #asr #whisper #self-hosted #multilingual-speech LinkedIn ↗

  • Grok 4.6 jumps to #7 on Code Arena — inside the error bars

    Arena

    Grok 4.6 landed at #7 with 1618 points, up from Grok 4.5 at #13 with 1553, putting it level with GPT-5.6 Sol xHigh at 1622 and Claude Fable 5 at 1627.

    Three models separated by 4-9 points is almost certainly inside the confidence intervals — a reminder of how little leaderboard rank means at the top of a board.

    #leaderboard #evals #model-release #coding-agents LinkedIn ↗

  • Three open releases from Hugging Face in one day

    Merve Noyan

    Qwen3.8 Max (A95B/2.4T, 1M context, #1 on the agentic Artificial Analysis index), Liquid's LFM2.5-VL-3B (small VLM, best in class across benchmarks, 32k context), and Cohere's North Micro Vision (small VLM at native resolution).

    Frontier-scale and small-VLM open weights the same day. The small end is where local speech-plus-vision pipelines start becoming feasible.

    #open-weights #model-release #vlm #benchmarks LinkedIn ↗

  • Meta's first open weights since Llama

    Sebastian Raschka, PhD

    Muse Glimmer, a 30B multimodal open-weight LLM.

    Meta re-entering open weights changes the competitive floor for the 30B multimodal tier.

    #open-weights #model-release #multimodal #meta LinkedIn ↗

  • Qwen3.8-2.4T compressed 91% via selective 1-bit quantization

    Daniel Han

    4.9TB down to 397GB through dynamic 1-bit quantization that selectively quantizes layers, runnable on 410GB+ of RAM or VRAM. Claims it still rivals GPT-5.6 Sol on many benchmarks.

    Selective-layer 1-bit is an aggressive claim. Benchmark parity at -91% size is the part that needs independent checking, not the compression ratio.

    #quantization #inference-optimization #open-weights #gguf LinkedIn ↗

  • Modal serves Qwen3.8 with a model-specific speculator

    Modal

    Qwen3.8-2.4T-A95B is available as a shared endpoint, served with a custom DFlash speculator tuned to the model's shape.

    Speculative decoding trained per architecture rather than a generic draft model — a serving pattern likely to spread.

    #speculative-decoding #inference-optimization #serving #modal LinkedIn ↗

  • Nemotron 3.5 Lightning on vLLM at 4x throughput

    vLLM

    Up to 4x higher throughput and 30% faster task completion aimed at always-on agents, via the OpenAI-compatible API. The model was trained on open datasets.

    Day-zero serving support plus open training data makes this one of the more reproducible agent-serving stacks currently shipping.

    #vllm #inference-optimization #throughput #agentic-ai LinkedIn ↗

  • NVIDIA adds a fourth projection alongside Q, K, V

    Akshay Pachaar

    The new projection predicts what the next layer will need — reported as 1.7x faster decoding and +6.5 points on long-reasoning accuracy.

    A small architectural change claiming both a speed and an accuracy win is unusual enough to warrant reading the paper rather than the summary.

    #architecture #attention #inference-optimization #nvidia LinkedIn ↗

  • Alibaba adopts OSU's prompt-injection benchmark for Qwen-CUA

    Huan Sun (OSU NLP)

    RedTeamCUA — a benchmark for computer-use agent robustness against indirect prompt injection — was adopted by XLangNLP and Alibaba Qwen for evaluation in the recent Qwen-CUA release.

    Third-party adoption by a major lab is how an adversarial benchmark becomes a de facto safety eval for computer-use agents.

    #agent-eval #prompt-injection #computer-use-agents #benchmarks X ↗

  • Grok 4.6 trained on xAI's own model-development work

    Yiwen Yuan (xAI)

    Her team built the training and evaluation stack that let the model learn from work accelerating model development itself — production inference and kernel optimization included.

    A concrete account of self-referential harness design, where the eval stack is built around the lab's own engineering workloads rather than public benchmarks. Worth reading against the AI4AI paper above.

    #eval-harness #model-training #grok #kernel-optimization X ↗

  • LlamaIndex publishes ExtractBench

    Jerry Liu (LlamaIndex)

    A 36-page arXiv whitepaper on what they pitch as the most comprehensive schema-guided, real-world document extraction benchmark.

    Worth reading for how they handle schema-guided grading and real-world document variance — the same design problems as structured-output eval.

    #benchmarks #information-extraction #eval-methodology #arxiv X ↗

  • A quantized inference framework teased; DeepSeek ships quietly

    Tim Dettmers

    An upcoming framework for running large models locally, posted alongside Cline's report that DeepSeek quietly shipped V4-Pro 0813, up 15.8% on Terminal Bench over their April preview.

    Two things at once: a forthcoming stack from a leading quantization researcher, and a silent frontier release with a concrete benchmark delta attached.

    #quantization #inference-optimization #model-release #terminal-bench X ↗

  • Upstage's Solar Pro 4 scores 42 on the AA index

    Artificial Analysis

    A new proprietary flagship reasoning model from the Korean lab, replacing Solar Pro 3 from April 2026.

    A useful data point on how non-US labs track on a standardized composite index rather than on self-reported numbers.

    #model-release #leaderboard #reasoning-models #benchmarks X ↗

  • Google proposes Agent Plugins as a portability standard

    Google Cloud Tech

    An open vendor-neutral standard for packaging Agent Skills together with the MCP servers they depend on — build it once, use it everywhere.

    If it lands, it changes how agent tooling gets distributed across vendors, which matters for anyone maintaining skills against a single harness today.

    #agent-skills #mcp #agentic-ai #standards X ↗

Briefly noted 15

  • Rohit Thotakura (Deepgram) (LinkedIn) — More on Flux: it starts speaking in as low as 80ms and carries tone, pacing and emotion across the whole session rather than per-utterance.
  • Daniel Han (LinkedIn) — Unsloth Desktop — the first open-source desktop app to both run and train models locally on Mac, Windows and Linux, supporting MLX, diffusion image and video, audio and GGUF, with sandboxed code execution and self-healing tool calls.
  • Unsloth AI (LinkedIn) — A 2-bit Nemotron 3.5 Lightning GGUF ran tool calls nonstop for 10 minutes on 22GB of VRAM, citing 80+ websites and executing code — a practical data point on aggressive quantization surviving long agentic runs.
  • vLLM (LinkedIn) — Ling-3.0-tiny from Ant Ling is open-weight in BF16, FP8 and INT4 with just 1.3B active parameters per token, scoring 25 on the Artificial Analysis Intelligence Index, with day-0 vLLM support.
  • Kwindla Hultman Kramer (LinkedIn) — Experiments in UI for AI agents — driving visual UI elements from voice agents, from the maintainer of Pipecat.
  • Dr. Alvaro Cintas (LinkedIn) — LangChain open-sourced Deep Agents Code, a model-agnostic CLI coding agent with isolated sub-agent context windows, a pluggable filesystem and human-in-the-loop.
  • Daniel Han (LinkedIn) — Unsloth for AMD — train and run 500+ models on Radeon, Instinct and Ryzen with as little as 3GB of VRAM.
  • Sebastian Raschka, PhD (LinkedIn) — "Build a Reasoning Model (From Scratch)" is now available on Amazon.
  • Dylan Fox (AssemblyAI) (LinkedIn) — On why voice agents trigger a different trust response than text agents — decades of bad IVR trees conditioned people to disengage the moment they hear "I'm an AI". (Disclosure: I work there.)
  • Qwen (X) — Teased a 27B model landing this week.
  • Steve Brunton (X) — Free PDF of his new Optimization Bootcamp book, also available from Cambridge University Press.
  • Subho Ghosh (X) — Working through the Colfax Research NVFP4 GEMM post and flagging a confusing point in its setup — low-level FP4 kernel territory relevant to quantized inference.
  • elyasbuilds (X) — Uses Pangram's AI-text detection research to argue who is actually winning the AI race, by measuring model-generated text share in the wild.
  • ziv_ravid (X) — Argues Bayesian learning is (almost) all you need for recursive self-improvement.
  • Sachin and Adam (X) — Long-form profile of Fortell, a $740M startup whose AI hearing aid has a waitlist — consumer-facing speech enhancement.

Past days

August 2026

MTWTFSS 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31