Peyash's Log

What's New — Friday, August 21, 2026

← latest digest ← 2026-08-19 2026-08-26 →

Papers 8

  • Evaluating Music Context Preservation: A Multi-facet Framework for Music Editing Systems

    UCSD · ISMIR 2026 ▲2

    MuseCPEval measures whether a music editing system leaves intact the parts it was never asked to change, with metrics grouped into four facets of musical attributes and validated both objectively and with human listeners.

    It flips audio evaluation from "is the edit good" to "what did the edit silently break". That collateral-damage failure mode goes largely unmeasured in speech enhancement and voice conversion too, where a system can improve its headline metric while quietly degrading something nobody scored. Packaging it as a diagnostic rather than a leaderboard is the right shape for that problem.

    #audio #evaluation #metric-design #benchmarks HF ↗

  • VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

    Huang, Tu, Han, Jiang, Wu et al. ▲5

    Introduces VAPref-10K — roughly 9k prompts and 10.3k pairwise human comparisons — plus VA-Judger-Bench, and trains a chain-of-thought reward model that scores jointly generated video and audio across separate quality dimensions. Used as an RL reward signal it improves generation quality in post-training.

    The transferable idea is decomposing a single binary preference label into per-facet judgments to get a denser reward signal from the same human effort. It is also one of the few preference datasets that treats the audio track as a first-class object rather than something scored incidentally alongside the video.

    #reward-models #llm-as-judge #audio #evaluation HF ↗

  • Training Leaves Traces: Centered Residual Signatures for Language Model Lineage Verification

    Aman Singh Thakur, Rayan Khoury ▲16

    Asks whether weights alone reveal ancestry between language model checkpoints. The method strips shared residual components and compares checkpoint-specific block structure, producing a symmetric lineage score calibrated against independent checkpoints.

    Reports perfect separation — AUROC 1.0 — of fine-tuned, LoRA-merged, pruned and quantized descendants from unrelated models, and claims robustness to deliberate weight-space laundering. A provenance result that falls out of residual-stream structure with no behavioral probe anywhere in the loop is a different kind of claim from the usual output-fingerprinting approaches, and a much harder one to evade.

    #interpretability #probing #model-provenance #residual-stream HF ↗

  • Looped Language Models Improve Compositional Tool Calling

    Popescu, Saez de Ocariz Borde, Liò ▲15

    Evaluates native and retrofitted looped (recurrent-depth) language models against non-looped baselines under identical training on API-Bank, BFCL and NESTful, focusing on multi-call workflows that have to carry intermediate state and dependencies.

    The gain lands specifically on compositional, dependency-aware chains and scales with recurrent depth, while isolated single API calls barely move. That separation is the useful part — it says what extra depth actually buys rather than reporting one blended benchmark number. Adaptive inference, spending extra loops only when needed, also beats fixed depth on the compute/quality tradeoff.

    #tool-use #agentic-ai #architectures #inference-optimization HF ↗

  • SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

    Tu, Chen, Huang, Mao, Zhang et al. ▲153

    Reframes video generation around whether a specified outcome is actually achieved while staying semantically grounded in a reference image. Ships SemComp-Data across six domains with a four-stage curation pipeline, plus a VLM-based harness scoring two metrics: Outcome Achievement and Generation Reliability.

    Splitting "did it do the task" from "did it stay faithful" into two scores instead of one blended quality number is the metric-design lesson, and current video models fail the combination rather than either half. Top-upvoted paper of the Aug 20 batch at 153.

    #benchmarks #evaluation #metric-design #video-generation HF ↗

  • Zetta: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence

    Ding, Mi, Wang, Liu, Cao et al. ▲186

    An embodied agent harness that evolves code-based runtime critics and recovery skills during execution while the base policy stays frozen, coordinated by three control loops running at action frequency, at rollout level for critic and recovery proposals, and at validated skill refinement.

    Existing agent harnesses reflect only after an episode ends; Zetta writes and validates new critic code mid-rollout. It reports 90.8% on LIBERO-Pro and 93.6% on RoboCasa with an 11.1x inference speedup, and was the top-upvoted paper across both days at 186.

    #agentic-ai #embodied-ai #self-improvement #rl HF ↗

  • Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

    Yang, Bian, Xiao, Vasconcelos, Li et al. ▲86

    Trains several parameter-decoupled models simultaneously with RL where each model's reward comes from its peers rather than from ground-truth labels. Cohort diversity — different families, sizes, rephrased samples — is the mechanism that keeps errors uncorrelated.

    Label-free peer reward normally collapses into mutual agreement; here heterogeneity is the explicit anti-collapse lever rather than an incidental detail. It still returns 3.0-8.6% average gains across seven text benchmarks and 2.3-7.2% across four multimodal ones with no ground truth anywhere in the loop.

    #rl #multi-agent #reasoning #reward-models HF ↗

  • ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models

    Li, Lin, Liu, Bian, Lam et al. ▲76

    Distills action-conditioned video world models into 1-, 2- and 4-step students through a staged pipeline of domain adaptation, teacher-forced causal training, consistency distillation and distribution matching, evaluated on Minecraft and FPS gameplay.

    The dual-path deployment is the interesting shape: run the 1-step student for real-time interaction, then optionally re-render the same trajectory at higher quality on replay, reaching reference quality with little trajectory drift. Top pick of the Aug 21 page at 76.

    #world-models #distillation #inference-optimization #real-time HF ↗

Tech radar 22

  • FA-Bench standardizes forced-aligner and timestamp evaluation across 12 aligners

    Wei Chu

    Released on the argument that most forced-alignment and word-level-timestamp benchmarks are not reproducible — differing phoneme sets, custom test splits and moving metrics make cross-model comparison close to impossible. FA-Bench standardizes the comparison over 12 aligners.

    Word-level timing is one of the areas where everyone reports numbers and almost nobody can compare them, because the evaluation choices move along with the model. A shared aligner baseline is the precondition for saying whether an alignment change helped, and the reproducibility critique it opens with is one most in-house timestamp harnesses would also fail.

    #forced-alignment #word-timestamps #benchmark #reproducibility LinkedIn ↗

  • A 0.63M-parameter verifier matches an off-the-shelf 7B

    Alexander Doria (Pleias)

    Yannick Detrois pretrained a 19-model ladder from scratch on SYNTH synthetic data, down to 70k parameters, to find where verifier capability actually saturates. A 0.63M-parameter verifier — roughly two minutes of training on a single H100 — matches a 7B model on specific verification tasks.

    The strongest eval-budget result of the day. If judge capability saturates this early on narrow tasks, paying frontier-model prices per grading call is a choice rather than a requirement, and the ladder shows roughly where it stops being one.

    #reward-models #llm-as-judge #scaling-laws #synthetic-data X ↗

  • Shuffle the task order and self-improving agents stop improving

    DAIR.AI

    A re-evaluation of memory-based self-improving agents. Adding two things prior work skipped — multiple runs to measure variance, and randomly shuffled task orders — erases much of the reported gain. The default task ordering had been imposing an implicit curriculum the results were riding on.

    The cleanest statistical-rigor result of the day, and it generalizes past agents: any sequential harness that runs once in a fixed order is measuring the order as much as the system. Both fixes are cheap, which makes their absence the interesting part.

    #eval-methodology #agent-evaluation #variance #self-improvement LinkedIn ↗

  • A 0.6B open-weights model for on-device transcript post-processing

    superwhisper

    S1-mini, their first open-weights release: 0.6B parameters, post-processing speech transcripts entirely on-device.

    A tiny open-weights cleanup model sitting downstream of ASR is the cheap baseline that larger speech-LLM pipelines should be measured against. If 0.6B on-device closes most of the formatting and disfluency gap, the interesting question becomes what the bigger model is actually buying.

    #speech-llm #asr #open-weights #on-device X ↗

  • Krisp Voice Isolation 2.5 strips competing speech, not just noise

    Sumanth P

    A pre-STT layer that removes overlapping speakers rather than background noise. The argument: WER spikes when two people talk at once, and conventional noise filtering cannot help because the interference is itself speech.

    Overlapping speech is a known WER failure mode that most eval sets under-represent, so it disappears into the average. A pre-STT separation layer is both a concrete lever and an argument for carving overlap out as its own benchmark slice — the gain is invisible until the slice exists.

    #source-separation #speech-enhancement #overlapping-speech #wer LinkedIn ↗

  • flashvad: an open VAD with explicit stable start/end events

    Himanshu Maurya

    A voice activity detector built for real-time calls: reads audio in 10 ms steps and emits a speech probability plus stable speech-start and speech-end events for interruption, barge-in and silence handling.

    Endpointing quality is what users actually perceive as streaming latency, and it is usually measured badly or not at all. An open VAD that commits to stable boundary events gives that measurement something to compare against.

    #vad #real-time-transcription #endpointing #open-source LinkedIn ↗

  • vLLM makes speculative verification adaptive, and publishes the acceptance curve

    vLLM

    vLLM now decides adaptively how much of a speculative draft to verify each step instead of requiring a fixed draft length. On DeepSeek-V4-Pro-0813 the first token of a 7-token draft survives verification over 70% of the time while the last survives under 10%; one configuration — adaptive verification at 7 speculative tokens — holds the Pareto frontier across concurrency levels.

    It removes a knob that previously had to be retuned per traffic pattern, and the per-position acceptance curve is the evidence rather than the marketing. That curve is also the reason fixed draft length was always the wrong abstraction.

    #vllm #speculative-decoding #inference-optimization #serving LinkedIn ↗

  • DSpark draft models ship with a quality-neutrality claim up front

    Liquid AI (via Piotr Mazurek)

    Draft models for LFM2.5-1.2B-Instruct, LFM2.5-2.6B and LFM2.5-8B-A1B, adding a speculative-decoding path at minimal memory cost with unchanged output quality — up to 3.18x throughput on H100 (LFM2.5-8B-A1B on MATH500 going 428 to 1362 tok/s) and up to 2.87x elsewhere.

    Published draft models rather than a described technique, with quality-neutrality stated as the headline. That is the claim that actually has to hold for speculative decoding to be safe in a serving stack, and stating it plainly makes it checkable.

    #speculative-decoding #inference-optimization #model-release #throughput LinkedIn ↗

  • CollabSkill ships the trajectories, not just the scores

    Yijia Shao

    Released the collected agent trajectories alongside the official CollabSkill benchmark, a human-agent collaboration measure built into Co-Gym, extending earlier work on which agents collaborate best across non-software knowledge work.

    The trajectory release is the part that matters. Publishing the raw traces is what makes independent metric design and contamination checking possible at all — most benchmarks publish a number and keep the evidence.

    #benchmark #agentic-ai #human-eval #dataset-release X ↗

  • Hawkeye uses expert unit tests as the supervision signal for kernel porting

    Arya Tschand

    Argues software support, not silicon, is the bottleneck for new ML chips. Hawkeye gives coding agents hardware awareness through a minimal taxonomy of expert-authored unit tests, enabling autonomous kernel porting across workloads, GPU architectures, vendors and precisions — BF16, FP8, NVFP4, MXFP4, covering GEMM and linear attention.

    The eval design is the reusable half: a small taxonomy of expert-written unit tests standing in for dense supervision is a pattern that transfers well beyond kernels.

    #gpu-kernels #inference-optimization #quantization #coding-agents X ↗

  • A Q3 quant beating a Q4 that is 4.5GB larger

    outsource_

    Benchmarked Unsloth's Dynamic V3 GGUFs on a single 4090, all runs on the same box, and reports a Q3 quant outperforming a Q4 carrying 4.5GB more weight.

    Same-hardware control is what makes this worth reading at all, and the result inverts the quant-tier heuristic most serving decisions lean on. Worth replicating before it gets cited as settled.

    #quantization #gguf #benchmark #inference-optimization X ↗

  • SeedVox puts a JEPA world model in front of speech generation

    Vincent Pollet

    Applies Joint Embedding Predictive Architecture world models to speech generation, on the argument that current TTS guesses emotion and rhythm token by token while a global predictive representation shapes delivery before generation begins.

    A structurally different answer to prosody than longer context or more tokens — and the same question audio-language models face on the understanding side: whether global intent is represented anywhere, or only inferred from local statistics.

    #tts #jepa #prosody #speech-generation LinkedIn ↗

  • Scaling laws that simulate the whole trajectory, not the endpoint

    Larry Dial (Marin)

    Reports that at Marin, scaling laws let them simulate an entire training trajectory rather than predict a single final-loss point, and that this became decisive in their workflow.

    Same extrapolation problem as forecasting a large benchmark run from small ones — and a curve shows where the estimate breaks down in a way a single predicted point never can.

    #scaling-laws #training-dynamics #eval-methodology X ↗

  • Context-aware PII redaction over voice transcripts

    LiveKit

    Strips personal information from voice-agent transcripts and audio recordings before storage, covering 41 PII types and using conversational context to detect sensitive spans.

    Context-dependent span detection over transcripts is a speech-understanding task with a genuinely awkward tradeoff: over-redact and the transcript stops being usable as evaluation data, under-redact and it cannot be retained at all.

    #pii-redaction #voice-agents #transcripts #speech-understanding LinkedIn ↗

  • Agents doing post-training lock in their strategy at step one

    Elvis S.

    A paper analyzing a large corpus of public post-training trajectories to test whether agents can post-train other agents. The finding: the agent fixes its training strategy at the very first step and spends the remaining budget on local adjustments. An experience-driven scaffold lifted execution by about 12.6 points.

    First-step lock-in is a failure mode reported with a measured mitigation, which is rarer than it should be. It also describes most automated tuning loops accurately — the search looks broad and is actually local.

    #agentic-ai #post-training #eval-methodology #scaffolding LinkedIn ↗

  • NVFP4 versus MXFP4, worked through a 32-element tensor

    Lewis Won

    Hands-on quantization examples for FP8, vanilla FP4, MXFP4 and NVFP4 over the same small tensor, surfacing the design considerations behind each format.

    Low-bit float is becoming the serving default, which means quantization-induced quality regressions start showing up in benchmark deltas. Worth being able to reason about numerically before attributing a metric move to the model itself.

    #quantization #nvfp4 #mxfp4 #inference-optimization LinkedIn ↗

  • How Claude's watermarking is supposed to work

    Sebastian Raschka

    Illustrates the scheme from the released materials: it exploits positions where multiple next-token candidates score similarly and biases the choice among them.

    A sampling-level intervention that rides on near-ties in the next-token distribution — which is also a quiet statement about how much slack sits in those distributions in the first place.

    #watermarking #llm-decoding #nlp-research #provenance LinkedIn ↗

  • Kimi's vendor verifier certifies vLLM for accuracy, not speed

    vLLM

    vLLM is now officially verified by Moonshot AI's Kimi Vendor Verifier for serving Kimi K3 at full accuracy, covering vision, 1M context and agentic coding paths.

    An eval-governance pattern worth noting: the model vendor certifies that a serving stack reproduces reference outputs, rather than leaving accuracy drift to be discovered downstream. That is a different contract from a throughput benchmark.

    #vllm #accuracy-verification #kimi-k3 #serving LinkedIn ↗

  • Mixture-of-Transformers throughput work landing in SGLang Omni

    Chayenne Zhao

    Extensive optimization of MoT performance in SGLang Omni, quoting SenseTime's SenseNova U1.5 Lite release.

    SGLang Omni is the multimodal serving path, so MoT throughput work there lands directly on anyone serving audio-language or speech-LLM models through SGLang.

    #sglang #inference-serving #mixture-of-transformers #multimodal X ↗

  • The case that continual learning goes in-context

    Ryan Boldi

    Argues continual learning is soon going to be largely in-context, pointing at recent work led by Jyo Pari.

    If domain adaptation moves from finetuning runs to context construction, the evaluation setup changes shape entirely — you stop comparing checkpoints and start comparing contexts, and very little existing tooling is built for that.

    #continual-learning #in-context-learning #nlp-research X ↗

  • DeepSeek v4 Flash resets the cost floor for harness runs

    Jay

    A long thread on what has happened since launch — speed, pricing and free access. It ran as a two-day trend on X with over 1,500 posts.

    Grader and judge calls are a per-run cost line in any evaluation harness. A fast, free, frontier-adjacent model is worth pricing in before committing to paid graders.

    #model-release #deepseek #inference-serving X ↗

  • Mojo is now open source

    Phil Eaton

    Modular open-sourced Mojo.

    An alternative to CUDA and Triton for custom kernels is now inspectable and forkable, which matters the next time a speech model needs a hand-written kernel.

    #mojo #open-source #gpu-kernels #inference-optimization X ↗

Briefly noted 17

  • Hervé Bredin (LinkedIn) — A reflection from the creator of pyannote speaker diarization on leaving academia to co-found pyannoteAI, after a decade building the open-source toolkit.
  • Dylan Fox (LinkedIn) — Pam's voice agents — nearly 1000 car dealerships, $200M+ in generated revenue, 20+ years of call time — moved to Universal-3.5 Pro after in-house evals showed it 3x better than their prior model with contextual prompting.
  • Javier Medel (LinkedIn) — NVIDIA released Cosmos3-DROID on Hugging Face: 707 GB of real-world robot training data, 71,907 episodes, 22.4M+ frames.
  • Minje Kim (LinkedIn) — IEEE Audio and Acoustic Signal Processing launched an AASP-managed Google Group as a general communication channel for the audio research community.
  • Tom Shapland (LinkedIn) — audiochatty.com launched — start a voice call when Claude Code finishes a turn to hear what it did and dictate the next step.
  • Bahushruth CS (LinkedIn) — Abliterated DeepSeek V4 Flash, following a 1M-download abliterated Qwen: refusal rate dropped from 68% to 3%, reported with KL divergence figures against the base model.
  • Gabriele Berton (LinkedIn) — An explainer on the modality gap in VLMs and how contrastive learning works — output embeddings of a randomly initialized transformer cluster tightly together even after normalization.
  • Andreas Horn (LinkedIn) — Cites Artificial Analysis measurements showing the same task costing $2.73 on one frontier model and $0.02 on another at comparable Intelligence Index, arguing against single-default routing.
  • Ryan Codrai (LinkedIn) — TurboVec 1.0 shipped on PyPI and crates.io with 2x faster search, holding a 10M-document vector corpus in 4 GB versus 31 GB as float32.
  • Mitko Vasilev (LinkedIn) — Reports 2,613 tokens/sec running Qwen3.8-27B across 4 GPUs in a Z8 Fury G5 workstation after a weekend of kernel and inference-engine tuning.
  • Nikolai Yakovenko (X) — NVIDIA licenses poolside for roughly $6B and hires 109 of their staff in a rare non-acquisition deal.
  • Tejas Bhakta (X) — Solo-dev inference provider Morph hit a $6M run rate as a one-person company; first inference provider live with Kimi K3 without early weights access, with 250 customer Slack Connect channels.
  • Jeffrey Huber (X) — Chroma announced Foundation, retrieval infrastructure described as the result of three years of build-up.
  • DynamicWebPaige (X) — Notes Gemini 3.7 Flash sitting alone on the cost-performance Pareto frontier at $0.25.
  • bun (X) — Bun 1.4 released — over 2,900 GitHub issues fixed, 1,517 tests added from the Node.js test suite, and lower idle resource use.
  • Dean W. Ball (X) — Launch of the AI Futures blog at OpenAI, with multiple contributors planned.
  • Sydney Runkle (X) — Built a browser agent combining Browserbase's Stagehand with LangChain's deepagents to test complex web navigation.

Past days

August 2026

MTWTFSS 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31