11 comments

  • _pdp_ 59 minutes ago
    This is the plasma TV phase of local AI.

    Early, heavy and priced premium. I estimate it will take ~5 years for most practical AI to run on our own machines, with a few super brains left in the cloud.

    I have been putting some effort into making local AI as consumer friendly as possible on macOS with this project https://github.com/pdparchitect/noodle/ but I will be the first to admit that the hardware you need is beyond most people's budgets. Hell, perhaps beyond most enterprises comfort levels too.

    For now, subscriptions are the most practical way in.

    • oytis 47 minutes ago
      Respectfully disagree. The bitter lesson has so far never failed us. And if in 5 years we'll be able to run models as capable as current SOTA on ordinary consumer hardware, imagine how much more powerful the models in datacenters would be
      • random3 4 minutes ago
        I actually believe it's going to be the other way around. Model progress is towards more efficient models, and mainly better compression, while the overall volume of data relevant to models' use-cases isn't increasing. In fact better models shouldn't depend on having all the data models have today either and should work efficiently without requiring all the factoids models encode today.

        Provided better models are going be available, reliance on models as a service should decrease considerably.

      • mycall 43 minutes ago
        Hopefully the point is that local models will be good enough for most day-to-day needs. They need more coherence and less hallucinations but that might happen within a year.
        • oytis 38 minutes ago
          Okay, makes sense, disregard then. I am a dinosaur whose day-to-day needs do not intersect with any AI usage
      • _pdp_ 41 minutes ago
        No doubt. The models in data centres will be ginormous, and impractical for everyday work, for the same reason nobody runs their spreadsheets on a supercomputer.
        • unrented7977 26 minutes ago
          That's a matter of perspective. Almost every computer used to run spreadsheets is orders of magnitude more powerful than that task actually requires. Compute got so cheap that one could reasonably afford to run spreadsheets on a supercomputer.

          Don't forget that your phone is several times faster than the world's best supercomputers just a decade or two ago.

          • dofm 2 minutes ago
            > Don't forget that your phone is several times faster than the world's best supercomputers just a decade or two ago.

            A decade? Not even close to true in the sense of the applications supercomputers are designed to do.

            A20 Pro in the iPhone 18 Pro, 2026: a shade under 3 TFLOPS.

            Even two decades ago: IBM BlueGene/L, 2006: 280 TFLOPs, 360+ peak.

            You have to go back three decades, to 1996 to find a machine it is "several times faster" than — ASCI Red in its first configuration.

            For sure, AI compute is an area where a phone has advantages. And, ironically, single core compute, because then ASCI Red is just one Pentium Pro. But ASCI Red had a terabyte of RAM, calculated in true 64 bit floating point, etc.

          • wolvoleo 11 minutes ago
            And Excel still struggles with a few million lines of spreadsheet lol.
    • mycall 45 minutes ago
      Theo@t3.gg has an interesting take that open source models won't catch up to frontier models, at least in the foreseeable future, as there are major roadblocks in place today that weren't there less than a year ago.

      [0] https://youtu.be/B2rFx4Fy1us

      • jazzypants 10 minutes ago
        I can already run models equal to Opus 4.6 on my freaking 3 year old MacBook air (m3 24gb). I don't need my local model to be much better than that.

        Like many others here, Theo has a vested interest in my experience not being real today. I don't really care what he has to say because he's wrong.

      • thereitgoes456 38 minutes ago
        Of course they won’t catch up. But they don’t need to. If they get to Gemini Neon / Opus 5.5 level, even a little below, it’s over.
        • hatefulmoron 27 minutes ago
          I'm hugely pro-local LLM, but I'm a little suspicious of this. People used to say the same thing about Opus 4.8 or 4.5 (GPT 4 as well?).

          Thing is, if you want to have a coding partner to implement things, the local models today are enough. But the frontier of what's possible is growing outwards to include things like superhuman code review, debugging, optimization, and one-shot program implementation.

          Who's to say, by the time we have a local equivalent to Neon/Opus 5.5, that we have a whole bunch of new tasks that the massive models are capable of that we now need to be replicated in local versions before we make the switch?

          • thereitgoes456 1 minute ago
            You could definitely be right. But, “it’s been true before” isn’t a convincing argument; it’s essential that the space of tasks that Opus can do meaningfully better than local models stays large. There’s only so good you can get at the common 90% of coding.
        • ladberg 29 minutes ago
          People keep saying that and it keeps not being true because they get accustomed to the frontier models.

          I'm certain that once you can run an Opus 5.5 quality model locally, whatever frontier model you can use in the cloud will outclass it so much that there will be no point to even touching your local model from a get-shit-done point of view.

          (of course, maybe you want ZDR and offline access and fine tuning and thus have reasons to run it locally, but that's a different conversation)

          • 0x696C6961 9 minutes ago
            I almost never use fable, opus is good enough for me.
    • e12e 35 minutes ago
      I think "practical ai" is a bit of a frontier model, wrapped in a harness wearing a trench coat in this comment.

      We already have useful local LLMs for voice to text, text to voice, translation, transformation/summation (transcript to formatted meeting notes) ... Stuff which is certainly practical.

      But local ai is still trailing behind frontier models.

  • thangalin 1 hour ago
    Since Spring, I've been working on a locally hosted, containerized web application to narrate my audiobook using a full cast of characters (skip to 9m47s for a reading of the demo book):

    https://www.youtube.com/watch?v=Q9PKT6bdKnc

    All weights are either permissively licensed or free until fantasy-level monthly recurring revenue sets in. The system runs entirely on the following hardware:

        CPU: AMD Ryzen 5 7600 (Zen 4), 6 cores / 12 threads, AVX-512
        GPU: NVIDIA T1000, 8 GB GDDR6 (Turing, 896 CUDA cores, 50 W, no BF16)
        RAM: 96 GB DDR5
    
    Text analysis uses Qwen3.6-35B-A3B MTP (Q4_K_XL GGUF) for speaker attribution, voice descriptions (not voice audio), emotive directions, and grammar. Both Muse Glimmer and GRMR cannot compete with Qwen3.6 for grammar checking. I haven't tried Gemma4, but I suspect Qwen3.6 remains the champ. My hope is that Inkling is more accurate for grammar than Qwen3.6, which is being tested at time of writing.

    Voice design and cloning uses FireRedTTS3, OmniVoice, Qwen3-TTS, VoxCPM2, and Ming-3.0-tts-tiny. Most are approximately equivalent. For narration, nothing comes close to Drama 3 Expressive Medium fine-tune in terms of quality, consistency, or pacing.

    Unsloth's GGUFs have been absolutely instrumental in making the system a reality. Highly recommend using them.

    • teravor 17 minutes ago
      tbh that demo was terrible.

      while it's probably on the way to being something good I found that just a single good voice and a CPU-only PocketTTS is sufficient for a good experience. just make sure to process line by line, cut intelligently (periods, commas, quotes, etc) to fit into max_tokens and insert pauses in between lines.

      there is a transition point somewhere between kokoro 82M and PocketTTS 100M where it just suddenly becomes good enough. kokoro lacks the tiny human touch that makes prolonged listening-for-pleasure tolerable and PocketTTS has enough of it.

      you do need to find a good voice for PocketTTS, for some reason the default voices they chose to include are pretty bad. some people swear that the right voice by chance makes PocketTTS perform better due to some quirk of training or something.

      here is a demo of PocketTTS 100M with the default kokoro voice cloned (in practice, I take one from audible): https://vocaroo.com/19piBx5VT5qy (speed it up to taste, mine is 2x)

  • afarah1 53 minutes ago
    A suggestion for the calculator: allow the user to set a baseline VRAM usage instead of a fixed 1GB "overhead" when checking "Card also drives my monitors" - for me, /usr/bin/X sits at ~139MiB VRAM on nvidia-smi, and if I don't run Firefox, that is the only thing consuming VRAM.
  • junga 1 hour ago
    Set this up this weekend after AMD NPU support was announced for Unsloth. My ThinkPad x13 with the Ryzen AI 350 and 32 GB RAM is now serving my older GhostBSD ThinkPad running crush. Using Qwen3.5-9B (npu2) the quality of course is not comparable to the latest frontier models. Basically no result can be trusted. And 12 tokens per second are quite relaxing but it somehow feels 'right'.
  • bashhq_r 1 hour ago
    how much vram do you need before qwen stops feeling sluggish on a normal laptop
    • yread 1 hour ago
      I run Basalt with Qwen 3.8 flash next (default Basalt NVFP4 quant) on 5070ti (16GB VRAM) with 96GB RAM. Prefill ~2000tok/s, generation ~100 tok/s. Definitely doesn't feel sluggish but it's not a laptop
    • Retell15 1 hour ago
      You need more processor power.
      • jfreds 12 minutes ago
        Processor FLOPS is probably only relevant if you’re doing some form of model parallelism and using your CPU for a little extra speed boost (a lot more complexity for maybe like, 5-10% speed up I’m guessing). Otherwise, your CPU is really just issuing instructions to move memory around, which is bottlenecked by PCIe bandwidth, not processor FLOPS. Processor power (watts) is pretty irrelevant im guessing.

        The sweet spot for VRAM for local model hosting is 128gb (Mac Studio or DGX spark). That gets you qwen-coder next at like Q8, so not lobotomized too bad, at 50 t/s, and a generous context window.

        As with most things, the real answer is “it depends”. All the VRAM in the world won’t save you if your HBM bandwidth is low

      • unrented7977 24 minutes ago
        No, you need more memory bandwidth.
    • steeleduncan 1 hour ago
      which qwen model?
  • amelius 1 hour ago
    What do the local-llmers do to get an experience closer to Claude?

    Do you reuse Claude's plugins/skill files?

    Do you use a special harness or system prompts?

    • segmondy 1 hour ago
      You can't get experience like clause but you can get result like Claude for 90% of tasks with more effort.
    • CamperBob2 17 minutes ago
      I just use Claude Code with DeepSeek 4.1F 1017. Works fine. DeepSeek has its own harness, which I'm sure is also just fine, but using CC with it makes it easy to go back and forth between DS and Opus/Fable.

      The guidelines in claude.md apply equally to DS, which is another benefit to sharing the same harness. No need to maintain two separate policy files.

    • buckle8017 38 minutes ago
      pi.dev

      you don't like how the harness works? ask your agent to change it

      this works surprisingly well

    • comprev 1 hour ago
      OpenWebUI, OpenTerminal and OpenClaw are useful tools for using local models. There are dozens more tools available depending on your needs.
  • 0xnNe 12 minutes ago
    > What Do You Need to Run a Local AI at Home? Not much (other than the hardware).

    > I'm using an RTX 5090 with 32GB VRAM for this setup.

    $10K GPU. Yeah, sure, not much at all.

  • angry_octet 1 hour ago
    So pretty soon 10,000+ people could be running decent models on their home PCs with unsecured LAN access, connected to their residential modem? We won't just have residential proxy networks, we'll have residential ARTEX swarm clouds.
    • geraneum 1 hour ago
      What do exactly expect to happen and how is it worse running other people’s agents on our machines?
  • IsTom 1 hour ago
    Saying that 32GB+ VRAM is needed for Qwen3.6-35B@q4 is certainly an interesting statement.
    • steeleduncan 1 hour ago
      That leaves 10GB(+) for context and KV cache, which seems about right
      • IsTom 1 hour ago
        It's MoE and it runs on 3060 just fine.
    • baoooooooooooo 1 hour ago
      Yeah, was this written before strata?
  • poulpy123 1 hour ago
    Easy, you just need a 10 000€ graphic card !
    • olelele 52 minutes ago
      I have been out of the loop on these prices. In sane.
  • snappr021 1 hour ago
    Doesn’t sound to far off the simplexity chatted to ChatGPT and Claude yesterday.

    When entire operating systems and software stacks to run end to end business processes, including engineering is a reality, Bill’s word “optimise” is close to the ultimate fulfillment.

    There is too much compute in data centres, when multiple machines can be replaced with a functionally intelligent pi.