Fosstodon

0 posts0 participants0 posts today

RainI want to do a good write up in my README for PixelPolygot as one of my last touches but I need the damn <a class="hashtag" href="https://melonbread.dev/tag/rocm" rel="nofollow noopener noreferrer" target="_blank">#rocm</a> fork of <a class="hashtag" href="https://melonbread.dev/tag/koboldcpp" rel="nofollow noopener noreferrer" target="_blank">#KoboldCpp</a> to update so I can do some more testing with Qwen2.5-VL locally. Like it works with vulkan on the main branch but way slower than rocm. :notlikeblob:

Reverend ElvisDeepSeek and the big step forward Yesterday, with very little effort, I managed to get a working <a href="https://social.undeadnetwork.de/tags/DeepSeek" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeepSeek</a> model running on my computer. It felt like I was a super-rich tech oligarch, but I just have a simple casual gaming computer with a cheap 8 GB graphics card for playing a game or editing videos. A lot of people have a machine like that; it's below average. And it took me half an hour. Hauke explains here how anyone can do the same. <a href="https://tube.tchncs.de/w/cuNhJ5tRzdv4WRE2oZjCv3" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://tube.tchncs.de/w/cuNhJ5tRzdv4WRE2oZjCv3</a>You only need the DeepSeek-R1-Distill-Qwen model (32 billion parameters, approx. 19 GB)<a href="https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-32B-Uncensored-GGUF" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-32B-Uncensored-GGUF</a>, which is a very slimmed-down version that will also run on a home computer. For institutes, educational institutions and companies, there are also more powerful models, right up to the full model, which then really requires enormous resources. And you need a program like <a href="https://social.undeadnetwork.de/tags/KoboldCpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#KoboldCpp</a> <a href="https://github.com/LostRuins/koboldcpp" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/LostRuins/koboldcpp</a> which loads the AI model into the graphics card and, for a weak card like mine, offloads the rest to the CPU. Furthermore, a chat GUI is provided via http://localhost:5001. As I said, if you've ever used a console, it's really not rocket science.There will certainly be loud voices that want to ban DeepSeek. Racists and capitalists will shout that the Chinese cannot be trusted. But it is open-source. Under an MIT license. I also ran it offline. It runs 100% locally.What you could do with it! If smart people continue to train the models. Now everyone has an AI and the tech bros can pack up! Thanks Deepseek and thanks China. A huge gift for the open-source community and maybe for humanity.PS. And once again, the conspiracy theorists were right, claiming that commercial AIs are only so large and resource-hungry so that they are reserved exclusively for the rich and the people remain customers. That's over now.<a href="https://word.undead-network.de/2025/02/10/deepseek-and-the-big-step-forward/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://word.undead-network.de/2025/02/10/deepseek-and-the-big-step-forward/</a> <a href="https://social.undeadnetwork.de/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ai</a> <a href="https://social.undeadnetwork.de/tags/china" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#china</a> <a href="https://social.undeadnetwork.de/tags/deepseek" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deepseek</a> <a href="https://social.undeadnetwork.de/tags/FOSS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FOSS</a> <a href="https://social.undeadnetwork.de/tags/ki" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ki</a> <a href="https://social.undeadnetwork.de/tags/openai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#openai</a> <a href="https://social.undeadnetwork.de/tags/opensourceAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensourceAI</a> <a href="https://social.undeadnetwork.de/tags/opensoure" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensoure</a> <a href="https://social.undeadnetwork.de/tags/oss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#oss</a> <a href="https://social.undeadnetwork.de/tags/techbros" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#techbros</a>

Reverend ElvisDeepSeek und der große Sprung nach vorn Mir ist es gestern mit sehr kleinem Aufwand gelungen, ein funktionierendes <a href="https://social.undeadnetwork.de/tags/DeepSeek" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeepSeek</a> Model auf meinen Rechner zum Laufen zu bekommen. Hat sich angefühlt, als wäre ich jetzt superreicher Techoligarch, aber ich habe nur einen simplen Casual Gamer Rechner mit einer billigen 8 GB Grafikkarte, um mal ein Spiel zu spielen oder Videos zu bearbeiten. So einen Rechner haben viele Menschen, es ist unterer Durchschnitt. Und es hat mich eine halbe Stunde gekostet.  Wie das für jeden nachzumachen ist, erklärt Hauke hier. <a href="https://tube.tchncs.de/w/cuNhJ5tRzdv4WRE2oZjCv3" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://tube.tchncs.de/w/cuNhJ5tRzdv4WRE2oZjCv3</a>Man benötigt nur das Model DeepSeek-R1-Distill-Qwen (32 Mrd. Parameter, ca. 19 GB) <a href="https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-32B-Uncensored-GGUF" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-32B-Uncensored-GGUF</a> das ist eine sehr abgespeckte Version, die eben auch auf einem Home Rechner läuft. Für Institute, Bildungseinrichtungen und Firmen gibt es auch stärkere Modelle bis zum vollen Modell, welches dann wirklich enorme Ressourcen benötigt. und man braucht ein Programm wie <a href="https://social.undeadnetwork.de/tags/KoboldCpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#KoboldCpp</a> <a href="https://github.com/LostRuins/koboldcpp" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/LostRuins/koboldcpp</a> welche das KI-Model in die Grafikkarte lädt und bei einer schwachen Karte, wie bei mir, den Rest auf die CPU auslagert. Des Weiteren wird eine Chat GUI über http://localhost:5001 zur verfügung gestellt. Wie gesagt, wenn man schon mal eine Konsole benutzt hat, ist das echt kein Hexenwerk.Es wird ganz sicher laute Stimmen geben, die DeepSeek verbieten wollen. Rassisten und Kapitalisten die rufen werden das man den Schlitzau... den Chinesen nicht trauen kann. Nur es ist Open-Source. Unter MIT-Lizenz. Ich habe es außerdem offline betrieben. Es läuft 100% lokal.Was man damit alles machen könnte! Wenn kluge Menschen die Modelle weiter trainieren. So haben jetzt alle eine KI und die Techbros können einpacken! Danke Deepseek und Danke China. Für die Open-Source-Community ein riesengroßes Geschenk und vielleicht auch für die Menschheit.PS. und einmal mehr hatten die Aluhüte recht, die behaupteten, dass die kommerziellen KIs nur so groß und Ressourcen hungrig sind, damit sie exklusiv den Reichen vorbehalten sind und das Volk Kunde bleibt. Das ist jetzt vorbei.<a href="https://word.undead-network.de/2025/02/10/deepseek-und-der-grosse-sprung-nach-vorn/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://word.undead-network.de/2025/02/10/deepseek-und-der-grosse-sprung-nach-vorn/</a> <a href="https://social.undeadnetwork.de/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ai</a> <a href="https://social.undeadnetwork.de/tags/china" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#china</a> <a href="https://social.undeadnetwork.de/tags/deepseek" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deepseek</a> <a href="https://social.undeadnetwork.de/tags/FOSS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FOSS</a> <a href="https://social.undeadnetwork.de/tags/ki" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ki</a> <a href="https://social.undeadnetwork.de/tags/openai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#openai</a> <a href="https://social.undeadnetwork.de/tags/opensourceAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensourceAI</a> <a href="https://social.undeadnetwork.de/tags/opensoure" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#opensoure</a> <a href="https://social.undeadnetwork.de/tags/oss" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#oss</a> <a href="https://social.undeadnetwork.de/tags/techbros" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#techbros</a>

uvokTried a roleplay with an LLM which more or less turned into a story. I actually quite like it. Could have more action, though Not sure if chat mode was the right choice. Also, still no idea how to do multi character chats.<a href="https://nc.uvokchee.de/s/7deWiWPAmB63K5E" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://nc.uvokchee.de/s/7deWiWPAmB63K5E</a><a href="https://furry.engineer/tags/koboldcpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldcpp</a> <a href="https://furry.engineer/tags/koboldai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldai</a> <a href="https://furry.engineer/tags/llm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#llm</a>

uvokYeah, this is what a mean by "the model gets confused". Apparently KoboldAI/cpp can't distinguish the stop words anymore.The initial prompt was<pre><code>User: This is a roleplay between two animal characters, which I will define later. User: Please generate and describe an animal character, which you will play. Choose a mammal. Please prefix your messages by a newline and your characters name, like so: ' Bob: waves "Hi, how are you?"'. </code></pre><a href="https://furry.engineer/tags/koboldai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldai</a> <a href="https://furry.engineer/tags/koboldcpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldcpp</a>

uvokAnd apparently, you can't have two characters as "you"...<a href="https://furry.engineer/tags/koboldai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldai</a> <a href="https://furry.engineer/tags/koboldcpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldcpp</a>

uvokHm, I'm unsure whether I should instruct the model to reply with their characters name prefixed. I have no idea how much "automagic" the chat mode has inbuilt. I actually wanna get away from only seeing "KoboldAI" and "user".Also, changing "my name" in the chat mode settings seems to confuse the model?<a href="https://furry.engineer/tags/koboldai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldai</a> <a href="https://furry.engineer/tags/koboldcpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldcpp</a>

uvok<a href="https://furry.engineer/tags/koboldcpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#koboldcpp</a> has a nice interface. I kind wish they had a better explanation what's the difference between "chat" and "KoboldGPT chat".Also, I'm simply overwhelmed by the amount of models. Each of them needs to be treated / prompted differently, it seems.And then I still have problems with the model generating my character's actions / speech occasionally, and I don't know whether that's due to my insufficient prompting, or due to this "end token" (e.g. <|im_end|>) not being configured correctly, which seems to differ for some models again.<a href="https://furry.engineer/tags/llm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#llm</a> <a href="https://furry.engineer/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ai</a>

Ryan He (帳號已遷移到其他站台)把 <a href="https://pastwind.top/tags/koboldcpp" rel="nofollow noopener noreferrer" target="_blank">#koboldcpp</a> 以前景模式執行，打開 <a href="https://aihorde.net/" rel="nofollow noopener noreferrer" target="_blank">AI Horde</a> 的任務分派程式，很明顯可以感覺到早上的使用者明顯比下午多。然後 RP 跟 ERP 的比例大約是 1:4。 <a href="https://pastwind.top/tags/LLM" rel="nofollow noopener noreferrer" target="_blank">#LLM</a>

HabrКак запустить Mixtral на своём компьютереВсякий раз, когда выходит новая хорошая ИИ модель, Хабр наполняется вопросами "Как нам её попробовать" и неправильными ответами, будто нужно платить за какие-то сервисы или иметь железа на сто лямов. Поэтому я вновь напишу инструкцию, как запустить новейший mixtral-8x7 на обычных средних компьютерах.<a href="https://habr.com/ru/articles/781702/" rel="nofollow noopener noreferrer" target="_blank">https://habr.com/ru/articles/781702/</a><a href="https://zhub.link/tags/LLM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LLM</a> <a href="https://zhub.link/tags/Mixtral" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Mixtral</a> <a href="https://zhub.link/tags/KoboldCPP" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#KoboldCPP</a> <a href="https://zhub.link/tags/GGUF" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#GGUF</a> #18+

Recent searches

Search options

Administered by:

Server stats:

#koboldcpp