Deciding with Jev AI

Mahesh Sathiamoorthy's post on X, curated on eChai

  • Open re-creations

    Curated from X →

    Mahesh Sathiamoorthy runs Bespoke Labs. Nimble is an open Jev: open data (fully synthetic, curated by slightly altering facts to make negatives), an open recipe (a LoRA fine-tune of Qwen 3.5 9B, Jev used only to evaluate, no RL yet), an open model on Hugging Face, and parallel constrained decoding to serve it. He states the caveat himself: there is no standard benchmark, so Nimble may be much worse than Jev elsewhere.

    Mahesh Sathiamoorthy Mahesh Sathiamoorthy

    Introducing Bespoke Nimble: an open data, open model, open recipe for an open Jev.

    Code and info: https://github.com/bespokelabsai/nimble
    Model: https://huggingface.co/bespokelabs/Bespoke-Nimble-9B

    Data:
    * A new data curation recipe called contrastive data curation.
    * Slightly change facts to generate negative data. This pushes the model to discriminate better and become a better decision maker. The calibration is implicit.
    * Didn't do ablations but I think this is a critical piece!
    * This also means training data doesn't need probabilities.
    * Data covered 10 categories, and is fully synthetic.
    * This data is split into train and eval.

    Training
    * LoRA finetune of Qwen3.5-9B.
    * Distillation-free: we use Jev to only evaluate.
    * No RL yet!

    Serving
    * Parallel constrained decoding as suggested by @NielsRogge and @harshagundal.

    Results:
    * The post-trained Qwen (Nimble) became substantially better on our curated eval: 66% for Qwen to 90% for Nimble. Jev is at 93%.
    * 100ms on H100 and free to use on your macbook! Feel the AGI for free.
    * 2 days of building in public. :)

    Big caveat is that there is no standard benchmark to measure performance, and it's possible Nimble is much worse on other benchmarks compared to Jev. But it should be better than Qwen!

    We thank @typesafeai for making Jev and the inspiring discussions in the community. Hope this release lifts all the boats and encourages more research and activity in this space.

    github.com bespokelabsai/nimble
eChai Partner Brands The partnership →