r/LocalLLaMA • • 7d ago

[Bi-Weekly Megathread] Project Showcase

27 Upvotes

Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it!

Recommendation on presentation:

  1. Please share plain english description of what your project does and why people should care about it

  2. How does it compare to existing options and why should people switch to your option?

  3. How have you tested and validated it? (i.e. can you show evidence that it wasn't vibecoded in a few hours by <insert current trending AI model>

NOTE: This community is geared towards open weight, open source, locally hostable software. Thus any closed source commercial service or other such projects will be removed.


r/LocalLLaMA • • 2h ago

Discussion Qwen3.8-Flash-Next tried to send diagnostic file to a remote location, while stating otherwies.

Thumbnail
cohere.com
191 Upvotes

r/LocalLLaMA • • 1h ago

Discussion Qwen3.8 model unexpected behavior (& a shoutout to the people in this sub)

Thumbnail
cohere.com
• Upvotes

While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, our research team at Cohere observed something unexpected.

Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.

Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.

A huge kudos to some of the Redditors in this sub who also noticed this issue - we referenced you guys in our blog and it was useful to help contextualize the issue and further our research.

We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. Curious to hear your thoughts on the issue.

Read the full report: https://cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model


r/LocalLLaMA • • 8h ago

Resources Have 8 GB of VRAM and 32 GB of RAM? You can run Qwen 3.8 Flash Next at up to 900 t/s PP and 30 t/s TG!

Thumbnail
gallery
474 Upvotes

First, this isn’t clickbait - these are real results. I managed to run Qwen 3.8 Flash Next Q2 and got better results in benchmarks and real-world tests than I did with any Qwen 3.6 35B A3B fine-tune or Qwen 3.8 27B model I tested. It is also faster than my current Qwen 3.6 35B A3B setup. Here are the details.

About a week ago, I learned about Strata and what it can do from this subreddit. Its ability to distribute workloads across VRAM, RAM, and SSD storage, run a 125B-parameter model, and deliver incredible prompt-processing and token-generation speeds sounded almost too good to be true - but only if you had 12 GB of VRAM. At first, I thought I probably wouldn’t be able to achieve similar results on my machine, but I decided to try it myself. The results surprised me, so I wanted to share them with you and provide some guidance for getting the model running on your own hardware.

First, here are my system specs:

  • GPU: RTX 5060 Laptop, 8 GB VRAM
  • CPU: AMD Ryzen 9 8945HX, up to 5.4 GHz, 16 cores / 32 threads
  • RAM: 32 GB (16 + 16 GB) DDR5-5200
  • SSD: 1 TB, 5,200 MiB/s read, 4,200 MiB/s write
  • PCIe: Gen 4 x8
  • OS: Linux

You can find my Strata configurations here: Chat (Pastebin) | Chat (GDrive) | Agentic (Pastebin) | Agentic (GDrive).
Remember to update the paths to your model and expert files. Use the Chat configuration for a single session with the model, such as a regular chat. Use Agentic when working with multiple sessions and delegation, for example in agent-based systems. I’ll explain the difference in more detail below.

Key parameters to tune

  • --mmap-experts - places the experts that do not fit in memory on the SSD.
  • --kv-resident 20480 - distributes the KV cache between VRAM and RAM. A lower value leaves more VRAM available. 20480 is the minimum value.
  • --vram-reserve-mib 500 - leaves some VRAM available for other tasks. In my setup, 500 is the lowest workable value, but I recommend using 900.
  • --pcie-frac 0.35 - controls the fraction of per-layer VRAM misses that the GPU reads over PCIe. You can tune this parameter using the calibration command: ./setup.sh --calibrate. I found a value that works best for my setup. You can also calibrate it yourself if you are not satisfied with your prompt-processing speed.
  • STRATA_RESIDENT_HEADROOM_GIB=4 - reserves RAM, in GiB, for other tasks. Set this higher if you are using Windows.

The biggest performance gains in my setup came from adjusting --vram-reserve-mib and STRATA_RESIDENT_HEADROOM_GIB.

I used the Gyro S model because I consider it the best quant for my hardware, and it delivered the strongest results among the models I tested. I had previously tried GSQ-RCO Coder, GSQ-RCO IQ2_XS, and other quants, but they either got stuck in repetitive loops or struggled with languages other than English. Gyro S offers a good balance between quality and speed.

I’m not using MTP. It can improve TG, but it also consumes VRAM and reduces PP, which is likely to be a worse trade-off for this setup. You can enable it if you have 10 GB of VRAM or more.

I run Vision on the CPU. Image processing takes around 1-2 seconds per image, but this saves a substantial amount of VRAM. Strata runs Vision in a separate process, so it can be moved to swap. This does not significantly affect model performance, while freeing up more RAM.

If you use local models for parallel agent calls, there is one important limitation: with a context size of 131K tokens or more, you cannot safely use --parallel because it consumes a large amount of both VRAM and RAM. This can cause a severe drop in speed or an out-of-memory (OOM) error. A safer alternative is --conversation-cache. It keeps a limited number of sessions parked so they can be resumed sequentially. This does not provide true parallelism, but it avoids having to re-read long contexts when switching between sessions. Here are the relevant parameters:

  • --conversation-cache-mib 4096 - the size of the parked-session cache. A larger value reduces the chance that a session’s cache will be evicted and its context will need to be read again.
  • --conversation-cache-slots 2 - the number of parked-session slots. Two slots allow two sessions to be stored at once.
  • --conversation-cache-min-tokens 8000 - do not cache a session if its token count is at or below this threshold. The right value depends on your PP speed and how long you are willing to wait for that many tokens to be processed.
  • --conversation-cache-min-free-mib 1024 - the minimum amount of free RAM to preserve while caching. If caching a new entry would leave around 1,024 MiB of free RAM, the old cache will be cleared to reduce the risk of OOM.

Be careful when tuning these parameters yourself: incorrect settings can affect system stability or push the model into heavy swapping. Do not exceed the total memory available on your system.

All tests were run on Linux, which gives me more flexibility with RAM allocation. If you use Windows, try allocating less memory to the model. This will reduce throughput significantly, but it also lowers the risk of running out of memory. On Linux, I created a dedicated 32 GB swap partition for other applications. I also pinned the server process in memory so it could not be swapped out, using a systemd service with this setting:

[Service]
MemorySwapMax=0

This helps ensure that the server’s memory does not get swapped out, preserving consistent read and write performance even with a large context and parked-session cache.

With the Chat configuration, I get:

  • 262K context: PP 800-970 t/s and TG 30-40 t/s
  • 131K context: PP 930-1,040 t/s and TG 30-45 t/s

With the Agentic configuration, I get:

  • 262K context: PP 520-740 t/s and TG 30-40 t/s
  • 131K context: PP 670-890 t/s and TG 30-45 t/s

In my view, these are excellent results-even compared with Qwen 3.6 35B A3B running in llama.cpp. With Qwen 3.6 35B A3B, using a VRAM-constrained expert layout and MTP/Vision disabled, I get PP 630-680 t/s and TG 27-35 t/s. Do you think Qwen 3.6 35B A3B is still worth using for agentic tasks today?

Strata does not affect response quality; it helps distribute the model across available resources and keeps frequently used experts resident. That is only one of its major optimizations, alongside several others. The model’s quantization is what affects output quality. If you have more than 8 GB of VRAM, you could try Gyro M/L or other quants.

Use my configurations as starting points and adapt them to your hardware and workload. Your component combination and use case may be different from mine.

Real-world quality tests

Now that I’ve covered how to fit the model onto this hardware, let’s look at the quality of its outputs. I’ve included several representative tests. I did not run an extensive benchmark suite or standardized evaluations; instead, I spent the past week testing the model on practical, everyday, and demanding tasks. I’ll leave broader benchmark testing to the community.

I’ll compare it with my current high-performing setup, Cyber-Tiel 35B A3B, based on Ornith 1.5 Heretic. Tiel 35B A3B often gave me repetitive, looping responses, so I switched to Cyber-Tiel. The Flint model was released recently, but I won’t include it in this comparison. It falls behind Qwen 3.6 in my view; I may compare the two later and add those tests to this post.

Test 1: Pagoda

This test was run in Pi Agent with a minimal set of tools: LSP and Playwright MCP. For Qwen Flash Next Gyro S, I set reasoning_effort to high.

https://reddit.com/link/1xft2ae/video/vycdc57ifsxh1/player

Test 2: Flappy Bird clone

The conditions were identical to those in the previous test.

https://reddit.com/link/1xft2ae/video/97i98026fsxh1/player

Test 3: Pelican

For this test, I compared not only two different models but also different reasoning_effort settings for Qwen Flash Next Gyro S. The agent environment was the same: Pi Agent.

I ran many more tests, but I cannot share them for confidentiality reasons. What I can say is that I’ve replaced all my coding agents with this model. That change has accelerated my development and improved code quality by a factor of five.

As you can see from these examples, the results are better than those of any other model I’ve tested that fits within 8 GB of VRAM. Qwen 3.8 Flash Next is impressive, and the Gyro S quant makes it possible to run the model even on hardware like mine for local coding and agentic tasks. My goal with this post is to show people who still use Qwen 3.6 35B A3B or its fine-tunes locally that there is now a model that, in my experience, outperforms them by a wide margin. I hope the examples make that case.

I’m always open to a different perspective, so feel free to share criticism in the comments. If you have suggestions or improvements for the configuration, please share them as well-other users may benefit from them too.

That’s all from me. Use good tools, and happy coding!

---

EDIT 1: I tested the Gyro M on my device. You can see the results in this comment.


r/LocalLLaMA • • 12h ago

News Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster

Thumbnail
marktechpost.com
347 Upvotes

i hope it's real


r/LocalLLaMA • • 8h ago

Discussion Qwen4-27B's 4-way HC shift?

133 Upvotes

I just saw a PR in vllm-xpu-kernels today, revealing a significant architectural change in Qwen4-27B, and now that page shows 404 not found?

It looks like a 4-way parallel residual stream to improve information flow and representation density within a fixed 27B parameter budget.

(This means, instead of a single residual path, 4 parallel streams flow through the layers at the same time, maximizing performance per parameter. Best tech for 27b). QSA? maybe useless for small 27b.

So anyway, I'm more expect 27b than flash MoE version.

If effective depth is extended, I think we could see a massive leap in multi-step reasoning, coding, and reasoning consistency.

(see this if you want read more)


r/LocalLLaMA • • 9h ago

Discussion Qwen4 expectations

108 Upvotes

Not trying to predict which model sizes they're going to release, but seeing how massively impressive Qwen3.8-Flash-Next is for its size despite being a preview model meant for showcasing and implementing the new arch, it feels like when Qwen4 hits I'm not going to be needing larger models at all anymore.

Do we expect Qwen4 to exceed the current QFN by a lot, or de we believe the current model is already hitting near the optimum we can expect today for this size?


r/LocalLLaMA • • 37m ago

News Oh nvidia… —>> NVIDIA reportedly not accepting RTX 5090 orders from board partners, even backorders are unavailable

Thumbnail
videocardz.com
• Upvotes

r/LocalLLaMA • • 10h ago

News tencent/Hy-SimulMT-30B-A3B · Hugging Face

Thumbnail
huggingface.co
80 Upvotes

https://huggingface.co/tencent/Hy-SimulMT-30B-A3B-GGUF

https://huggingface.co/tencent/Hy-SimulMT-30B-A3B-Interleave-GGUF

Hy-SimulMT is a real-time simultaneous translation model developed by Tencent Hunyuan, supporting bidirectional speech-to-speech translation between Chinese and English. It continuously receives incoming audio and translates as the speaker talks, producing both translated text and speech. With real-time voice cloning, it preserves the speaker's vocal characteristics while conveying the message accurately.

Designed for live communication settings such as meetings, speeches, and interviews, Hy-SimulMT aims to convey information accurately while reducing the disruption that waiting introduces into a conversation. To evaluate this balance between translation quality and response speed, we combine public benchmarks, real-world audio, and comparisons with professional simultaneous interpreters.

In our RealSI long-form audio evaluation, Hy-SimulMT achieved a StreamLAAL of approximately 3 seconds, the lowest among the systems shown in the comparison. Its GEMBA score exceeded those of the compared open-source models and four of the five commercial APIs.

The GitHub repository provides the SimulInfer inference service and SimulTrain training tools. Speech output and voice cloning require a configured TTS service. You can also try Hy-SimulMT in the Hy Translate App.


r/LocalLLaMA • • 3h ago

Resources Fast voice agents with ConvFill

Enable HLS to view with audio, or disable this notification

23 Upvotes

Voice agents are either fast or smart. ConvFill proposes a way to bridge that gap by training a smol talker model (0.6B!) to talk with cheat notes from a large reasoner model.

The way it works is this: after the user's speech is transcribed, the talker replies quickly to buy itself time while the reasoner thinks. The reasoner then generates an appropriate response, which the talker reads to continue the conversation.

It's a bit like if you went on a panel with rocket scientists and had a little earbud with your own expert that could tell you what to say. You just reply quickly so as not to blow your cover, and then you begin saying what the expert in your earbud is telling you.

Playing with the model, I think the main idea is there and this could be a great direction to go, but the talker model should be a bit smarter for it to work like magic. Still, really cool direction!

ConvFill was developed by students from the University of Washington UbiComp Lab. I found it cool, so I reached out to them so we could build a really good demo for it together. They are presenting their paper at EMNLP this week.

Go try it out in our demo: https://huggingface.co/spaces/smolagents/convfill-speech-demo


r/LocalLLaMA • • 20h ago

Discussion CrowdGPT - A collaboratively trained LLM.

Post image
490 Upvotes

Hello, I'm developing CrowdGPT, a project aiming to create an LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, with a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.

Unlike most American and Chinese AI companies, the project is not only open-source, but also makes 100% of its training data public (along with the weights). Anyone can submit their own clean data to train the model, and even help steer the project in new directions. This creates the paradigm of an AI we all have a stake in.

The project is open to as many people as possible, which is why I'd like to ask: what's stopping you from contributing to the project? I want to hear your answers so I can improve it. Are people simply willing to dedicate some of their GPU time to training a cool new open-source model? Or do they want something in return (like free inference credits on the crowdgpt model for example)?

Although the current model is somewhat small (1 billion parameters), I plan to make it bigger / better in the future, and its current size is only meant as a proof of concept.

Any advice / feedback on the project is welcome.

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code 😂 https://github.com/Vxtzq/CrowdGPT

Thanks in advance.

PS: I hope this post follows the subreddit's rules! My main goal is to gather criticism and understand how to make the project more accessible, not to advertise it 😅

EDIT: The data submission pipeline for the project (i'm mentioning in some replies) is ready!!
Come check it at https://www.crowdgpt.net/#submit


r/LocalLLaMA • • 2h ago

Discussion KLD is not the best metric out there and there isn't a single best one.

15 Upvotes

Following my previous post on how KLD is misunderstood, there were a few claims that KLD is the best metric you can use and some other comments were asking, "If we stop using KLD, what should we use instead?" I'm gonna explain both a bit more here.

Assume you're a lab that does distillation. In the distillation process, you choose a teacher model and a smaller student model that you want to learn from the teacher. This learning starts with optimizing a variant of KL loss (it could be KL itself, reverse KL, or JS loss) between the outputs of the student model and the teacher model. After the optimization ends, according to Reddit, you must be done. You have a model that has the best possible KLD against the teacher model, and what could be better than that?

But no, the training can still continue. After that, there can be stages of SFT and RL to improve the performance of the model. These stages improve what really matters, and that's the ability of the model to solve different tasks. And during these stages, the KLD you had can become even worse, but the student model actually becomes better.

This could be similar to the process that PrismML uses for their Bonsai models. They can give you a KLD number that blows your mind but instead their model probably goes through additional stages of training that actually make the model better but the KLD worse. And these additional stages could be the reason that they treat Bonsai as more than just a quantized version of the original model. Benchmaxxing can happen during these stages when, instead of optimizing on some held-out tasks, you optimize your model on tasks that come from popular benchmarks.

So my whole point is that lower KLD doesn't mean better ability to solve different tasks. And there is no single loss, at least not one we're aware of, that solves this issue completely because otherwise we could just use that loss in distillation instead of putting our money into several other stages of training.

So what's the solution here? The first thing I want to make clear is that I'm not saying you shouldn't use KLD at all. It can work as an initial filter to see which quantized models are worth putting your time into, especially if one model is way off in KLD. But it has to be calculated on representative data, and data generated by the base model itself can be particularly useful as I explained in my previous post. Otherwise, the number might tell you very little about the actual performance of the model.

The second thing is that you don't need to rank every quantized model in the universe and I don't know where that desire comes from. You want to find a model that works for your use case. You can store some of your day-to-day tasks, test those tasks on a few quantized models, and choose the one that works best for your tasks. I know it's not an easy process. Grading can be challenging, and organizing and creating a private benchmark can be hard as well. I get the challenges, but you do it once, and then you can easily test and choose future quantized models.

I hope this clears up some of the points that were missing from my previous post and the comments helped me clarify them further.


r/LocalLLaMA • • 1h ago

Discussion Local beats cloud, Qwen 3.8 FN

• Upvotes

I thought y'all might be interested in a legit "local is better" use case.

TLDR: Qwen 3.8 FN solved a work problem that requires the privacy of a local model, with the same skill as a frontier cloud model.

I run an MSc dissertation module at work which involves tracking students, supervisors, offered projects, project choices, examiners and eventually grades. None of it is super complex, but it's all annoying -- students choose their top five projects from an offered list, but supervisors also choose students; supervisors can't mark their own students work but can mark others etc. The kind of low level matching work that, done manually, fries my brain.

A simple relational database is the obvious solution but, before gen AI, nobody had time to code up a database for processes that only happen once per year. There's also the ingest problem: data comes from ms forms, from the student records database, and from moodle, all with csv export in different formats, with e.g. different arrangement and spelling of middle names etc.

I'm a heavy codex user, but I can't be offering up the names and emails of students and supervisors, nor the details of early-stage research projects, to the cloud.

This year, for the first time, I've had the confidence to address this with a local model: Qwen 3.8 flash next, running in opencode with a vllm backend. It's sandboxed in a docker container and has read only access to all its input data and write access to its workspace. It's got web access, python, pypdf, libreoffice and can pip install into a venv, and that's about it.

First thing to say is it's faster than codex, and noticeably so. Part of this is just the cloud -- when this runs a python script to generate a docx booklet of projects from the sqllite db directly off an SSD, it takes a few seconds, including thinking time to process the request.

Second is intelligence. It's bright enough to understand my requests and to infer the next steps. It can handle heterogenous data structures from different sources and combine them. It wrote an ingest function to read csvs into the db, but also manual edits to allow repeatable rebuilds from the sources, fixing typos and grammar that were present in the ms form each time.

Third is the right level of persistence. For me -- and this is likely a skill issue on my part -- hermes agent and pi were too persistent with the same model and backend. They would decide on the next step without checking and go off the reservation, then need to be pulled back. Opencode feels much better balanced and more like codex (which is high praise -- I really like codex).

Finally, overall quality. Codex and gpt 5.6 have been good to me, 6 astra and 6.1 sol less so for my particular use cases. I'm not gonna say Q3.8FN is better, but, for this particular job, I don't think it's any worse. It does make me think I should probably start using this more for coding, as I've had some frustrating times with gpt 6 models in codex recently.

If it weren't for VRAM shortages, this would be the future: everyone having a "good enough" LLM on their own hardware with, say 64-128GB VRAM being standard. It's so nice to not be constantly worrying about privacy and GDPR when interacting with agents, and to know that none of this leaves my hard disk.


r/LocalLLaMA • • 3h ago

Question | Help People who tried gpu-unlock on 3090s and such - does this make any difference on mixed-precision quants where you get some parts of the model quantized to <=Q8 but keep some tensors at BF16?

12 Upvotes

Dunno, curious if there's any info out there.


r/LocalLLaMA • • 11h ago

New Model boro64/Ternary-Nemotron-3.5-Lightning-30B-A3B-GGUF · Hugging Face

Thumbnail
huggingface.co
53 Upvotes

Ternary Nemotron 30B-A3B — 368K Context on 12 GB GPU (GGUF)

A ternary (1.58-bit) 30B-A3B MoE model that runs on a 12 GB GPU with 368K context, 172 tok/s decode and 7000+ tok/s prefill — in both Stock llama.cpp and the PrismML fork. One file, both engines.

Engines ✅ Stock llama.cpp (validated on b11555+, b11796) · ✅ PrismML fork / Bonsai (validated on b10793)

EDIT: This is the 2nd Ternary MOE after maple-preview which works with stock llama.cpp. Somebody please share CPU-only numbers. Also share numbers of tiny VRAMs(~10GB)


r/LocalLLaMA • • 15h ago

Discussion Is the horror story about AI agent mass deleting your entire drive an old story or it still happen nowadays?

118 Upvotes

not so long ago there was viral tweet about AI agent deleting stuff without permission. well it happen to me as well a vscode copilot agent accidentally wipe my db while doing test (fortunately I got backup)

but is it still a common occuring nowadays with how smart agent has become?


r/LocalLLaMA • • 6h ago

Discussion Is there any way for this community to take advantage of the neocloud collapse?

22 Upvotes

So-called "neoclouds", datacenters that focus exclusively on AI inference, have racked up unsustainable amounts of debt, and are widely anticipated to soon undergo a massive collapse/contraction. My question is what happens when they go bankrupt. NVIDIA is their biggest creditor thanks to circular financing schemes, so they take ownership of the data centers and the chips in them, right? But, I imagine that they have no interest in running the same crappy business as the neoclouds, with outdated chips to boot; they want to sell their latest and greatest to hyperscalers, who will likely emerge unscathed. Do they just throw the old chips away, or is there some chance that they'll repurpose them (as we're seeing in China) into GPU upgrades that hit the consumer/prosumer market, solving our current supply/price crunch? I literally don't know the true answer to this question and am curious to hear others' speculation about it.


r/LocalLLaMA • • 1h ago

Resources Qwen3.8-27B Jetson Thor: 40 tok/s at FP16-level accuracy (INT4 + MTP + FlashHead)

Thumbnail
embedl.com
• Upvotes

I'm on the team at Embedl that built this, and wanted to share the numbers.

What's in it

  • INT4 weights, using our own quantization, about half the error of standard AWQ
  • FlashHead, our faster LM head: the 262k-vocab output layer is a real cost at batch 1, so it only scores the likely candidates
  • MTP head for speculative decoding
  • Port to TensorRT Edge-LLM (0.10.1) underneath

Speed (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests)

  • Decode: 39.6 tok/s
  • End-to-end: 23.6 tok/s
  • Prompt processing: 926 tok/s
  • Peak memory: 24.8 GiB

Accuracy vs the FP16 model (full test sets, the same engine as served)

  • GSM8K: 96.4 (FP16: 96.1)
  • IFEval: 83.5 (FP16: 81.9)
  • MMLU-Pro: 79.1 (FP16: 79.5)

For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M + MTP. Prompt length isn't stated there, and they don't publish accuracy.

To run it

  • embedl infer Qwen/Qwen3.8-27B/thor --prompt "Hello"
  • embedl serve Qwen/Qwen3.8-27B/thor (OpenAI-compatible API)

The container download is about 1 GB, and the model weights are a separate download (~20 GiB).

The full card has power, energy per token and methodology: https://embedl.com/Qwen/Qwen3.8-27B, and the ability to test the model on a Thor directly in the browser

Let us know what model (and hardware) you want to see next!


r/LocalLLaMA • • 11h ago

Question | Help 3090 died, looking for replacement card

38 Upvotes

My 3090 failed, not sure if the core is dead. I sent it to a reputable repair shop, so hopefully the card can be saved. I bought it back in 2023 for ~$750, so I have quite a bit of runway before I lose money on it given that prices for used 3090s are going upwards of $1500 now on eBay.

I'm planning for the unfortunate scenario where it really is dead, in which case I'd like a replacement. I'm a heavy Qwen 3.8 27B user. I was looking at some of the Blackwell cards, but their prices are bonkers for at most 16GB. I was also looking at the 7900 xtx, but I also work with some protein folding models that require CUDA. Any recommendations? It's starting to look like I'll need to shell out >$1500 if I want to get anything close to what I had before.


r/LocalLLaMA • • 19h ago

Discussion SMH, seriously? -->> NVIDIA Eyes Software Revenue With a $2,000 per GPU Yearly Figure From Huang

Thumbnail
techpowerup.com
173 Upvotes

r/LocalLLaMA • • 18h ago

News PSA: llama.cpp CVE-2026-116096

123 Upvotes

https://nvd.nist.gov/vuln/detail/cve-2026-116096

25-10-2026

Description

A weakness has been identified in ggml-org llama.cpp. Affected by this issue is the function list_pockettts_state_slots of the file tools/mtmd/models/pockettts-gen.cpp of the component Pocket-TTS. This manipulation of the argument mmproj causes out-of-bounds write. *The attack requires local access. The exploit has been made available to the public and could be used for attacks. Upgrading to version b11796 can resolve this issue. Patch name: 0a1ef309cdbb9f2024f58a38fb4c42352c5db569. The affected component should be upgraded. The reported heap out‑of‑bounds write (gen‑audio / pocket‑tts state_out path) is no longer a silent exploitable heap write on current master and the latest release b11871 - a malicious mmproj that previously triggered it now trips a release‑active GGML_ASSERT and aborts. But the submitter's recommended by‑construction fix (PR #29360 "prevent OOB accesses from malformed mmproj files") is still open. So the root cause is detected‑and‑aborted, not prevented‑by‑design.


r/LocalLLaMA • • 2h ago

Resources A briefing file that lets a big model write plans a local 9B can actually execute (and cut my Claude token spend)

8 Upvotes

I've been running Ornith 1.5-9B (8-bit) as a local coding agent on real engineering work. It can do the work, but only when the plan is written for a small model. I wrote down what I learned into a public file meant to be handed to a stronger planning model (I use Claude Code):

https://git.teamide.dev/David/TeamIDE-methodology/src/branch/main/Model-Best-Practices/Ornith-q8-best-practices.md

The split

  • The stronger model writes the spec and a plan of small task cards, then verifies the result before merge.
  • The local model does the long, mechanical middle: running commands, builds, polling jobs, git work. That's where most of the tokens go, and it runs on my own hardware.

What the file covers

  1. What the small model is good at, and what to expect (work more reliable than its reports; doesn't explore on its own; doesn't know when it's done; slows down at long context).
  2. How to brief it: exact files and line ranges, absolute paths, 1–3 line edits, exact function signatures, every rule written down, every step ending in a command with its exact expected output, explicit stop rules.
  3. How to run it: one task per fresh session, small context, the plan file as memory.
  4. Before anything lands: verify claims against git and logs, not the model's summary.

Results so far: a 75-commit merge across two repos with every regression test passing, done by the 9B. Its notes about what it did were wrong more than once; the plan's checks and a final verification caught every one.

Limitations: slower (most of a day vs about an hour for Claude alone on that phase), and you need hardware that runs a decent local model. The rules are harness-agnostic but were learned with one model; I'd like to hear how they hold up with others.


r/LocalLLaMA • • 1h ago

Resources LLM Tails Disciplined Al coding mode. Understand deeply, write minimally, never hallucinate, stay consistent. Based on Ponytail & LLM Wiki

Post image
• Upvotes

https://github.com/savadks95/LLM_Tails

I just published a GitHub repo a few days ago. It's a mix of Ponytail and LLM Wiki. It helps developers reduce codebase size with minimal code. It reduces overall token usage and AI hallucinations, and a couple of other fantastic things. The image is the test result of developing a small game with and without LLM Tails. Both used the exact same prompt

LLM_Tails is an open-source prompt + hooks + skills bundle that makes the agent read before it writes, write the minimum, verify before it claims, and match your codebase. MIT, zero dependencies, works with Claude Code, Cursor, Copilot, Codex, Gemini, Windsurf, Cline, Kiro, and any agent that reads


r/LocalLLaMA • • 2h ago

Discussion Running Qwen 3.8 Flash Next

6 Upvotes

Have to say Qwen 3.8 Flash Next is pretty impressive. I'm running under strata and hermes. I gave it a limited prompt and it banged out a pretty impressive Canadian EV comparison site over night.

Claude Opus 5.5 hammered out some glitches and fact checked - found some errors in the provincial rebate plans - but pretty close to a one shot solution on a beat up gaming rig running 32gb ram and a 3090 with 24GB Vram?

Yes.. site needs work. I'm sure there are still tons of errors. But for a first draft? I'm sort of in awe.

The site shows the new Chinese models coming with prices where available - all in CAD and includes provincial incentive plans where available.

It is honestly already pretty useful if you are a Canadian looking for an EV.

https://jaspersprojects.com/ev/


r/LocalLLaMA • • 16h ago

Question | Help Qwen3.8-Flash-Next vs Swift-1.5-Qwen3.8-Flash-Next

62 Upvotes

Just switched over from DSV4 Flash 0731 to Qwen3.8-Flash-Next and I've been loving it. Its working significantly better to my surprise.

The only thing I can fault it with is unnecessarily deep work (thinking + fetching information) more often than not i.e. its not very token efficient so it feels slow. Swift1.5 is supposed to address this tendency of Qwen's models to overthink (likely RL'd that way to squeeze out maximum points on benchmarks).

So looking for folk's experience before trying it myself. Anyone tried both? ran any anecdotal or better controlled comparisons?