Project NOMAD: Running LLMs Locally From a USB Drive
Yeah, you heard it right, man. We’re gonna use a flash drive — yes, a literal flash drive — to run a Local LLM, or simply an Offline AI.
All you need is a flash drive and a decent laptop. No graphics card? I gotchu, man. And believe me, whether you have a graphics card or not, it will work even on a potato PC. :)
Also, if you do have a graphics card, congratulations — you just unlocked the peak lvl. since your graphics card will take a lot of the load off the RAM itself. :D
Also, please don’t judge me. This is my first post here. I’m just doing some stuff to gain attention and complete my college assignment. T-T
Now, for the first step, open your laptop and grab a flash drive (btw, when was the last time you touched grass, brotha?? Just curious :D). Now, connect the flash drive to your laptop and format it. Before doing that, make sure you select exFAT as the file system. Refer to Fig. 1 and Fig. 2 for that. Make sure the flash drive is empty before formatting it, or u will lose all ur precious data. !! CAUTION: Do NOT format your operating system drive T-T. Double-check that you have selected the correct drive (flash drive/pendrive) before hitting format !!
After formatting, we move on to the next step towards our project — IUPAC for our project (don't know what IUPAC is? R u for real T-T). Rename the pendrive however u like, maybe "Apple" or maybe ur crush's name ;) just kidding. But for this project, I'm naming it "NOMAD". Now that we have a name for our project, what we need next is a structure, kinda like a skeleton. In our project, this skeleton is basically the folder system — how we're gonna organize everything. Now, open the drive we just formatted and create these folders in it, (engine,models,docs,chats,prompts,logs,
config). You can also refer to Fig. 3 for clarity:-
Wait... u don't know what a .bat file is either?? Bro, u must be living under a rock fr T-T.
Now, since we have all the folders ready, we r halfway done. Think of it like making a car — maybe a Bugatti or a Lamborghini. There r multiple steps involved, like making the outer shell/cover, headlights, engine, gearbox, etc.
So what we did right now was basically make the shell of our car. But obviously, a car can't run without its most crucial and important part — the "engine". And that's exactly what we're gonna set up next. :D
So here is a simple analogy: in a car, the engine provides power to the wheels so the car can run around the town (same as u do when hanging out with your friends :D). In our case, to run these Local LLMs, we need something that communicates with our operating system and our laptop's hardware so it can actually run the AI. There r multiple so-called local AI runtimes/tools (I’ve also added a link to a dev who explains these tools in more depth, so if u wanna know more, just click on the blue term) where u can run Local AI, like Ollama, Jan, LM Studio,and llama.cpp, etc. We r personally gonna use llama.cpp, since it's convenient, flexible, and maybe my personal favourite. :)
Now, see what these tools do is, when u run an AI through them, they create a localhost server and host your AI on it. This allows u to upload photos, files, view chat history, use custom prompts, and do all that good stuff. It’s way more convenient than using the terminal since it gives u much more control and flexibility over our project. And man, why wouldn't we use that? No one’s gonna sit there using the terminal like a grandpa, unless and until u have something like 2GB RAM and no graphics card. In that case, the terminal can actually be a good option. But I trust u guys have at least 4GB RAM in this RAM-consuming Windows 11 world T-T.
Jokes apart, these environments are basically what can give u the closest experience to using ChatGPT, DeepSeek, Gemini, etc., at least in terms of the user interface and overall experience. Now head over to the official llama.cpp page and scroll down for the download option, as shown in Fig. 4. :D
Now, the best part that I genuinely like about llama.cpp is that it works across so many different devices and operating systems — from Mac to Android, and from Linux to Windows. Like genuinely, man, there r a huge variety of users who prefer different operating systems depending on their use. I personally like Kali Linux, and why not? It's a favourite among cybersecurity enthusiasts like me { °-° }. Maybe someone likes Mac, while someone else prefers Windows, it all depends on the user. This is actually why I wanna salute llama.cpp for their hard work in optimising it for so many different users and devices. (2 minutes of silence for them T-T)
When u click on one of the download buttons, u will land on their GitHub page, where u will find multiple files available for download same as shown in Fig. 5,6 :)
Since we r developing this for Windows for now, I downloaded Windows x64 (CUDA 13) since I have an NVIDIA RTX graphics card. Don't worry if u don't have a graphics card — u can select Windows x64 (CPU) instead. Or, if ur device uses ARM architecture, select Windows ARM64 (CPU). Don't get confused with all these options T-T. Just upload ur system information to ChatGPT, and it can help u choose the version that is best suited for ur device. After downloading, we get a ZIP file. Extract it, and u will get a folder from it. Open that folder, press Ctrl + A, then Ctrl + X. Now go to ur pendrive, open the engine folder, and paste everything by pressing Ctrl + V. Make sure u paste all the files inside the extracted folder !! CAUTION: Don't paste the entire extracted folder inside the engine folder. Paste the files that are inside that folder directly into the engine folder !! , same as shown in Fig. 7, 8, and 9. :D
Alright guys, we did most of our work. Now we have our car's shell and engine. What we need next are the wheels of our project — the actual thing that will run our Local LLM on our devices. So, to do this, we can download models like Qwen, Gemma, DeepSeek-R1, and many other open-source models. Since we're currently not going to make an LLM from scratch — because that takes a lot of blood, sweat, mental health issues, and probably a few tears T-T — we're taking a shortcut by using already existing models. But I promise, in the future, I'll come up with a post on how to make and train an AI/LLM from scratch. It will be a cool project, I guess. :D Currently, many companies have already developed and released their own open-source AI models, and many startups have built their products using them. As a big brotha, I wanna tell u something — buying an AI subscription is sometimes like buying bottled air when there is already a lot of free AI around us :) Don't get me wrong, paid AI services can offer better models, convenience, cloud computing, faster speeds, APIs, and features. But for many use cases, u can actually run powerful open-source AI models locally for free instead of paying a monthly subscription. A lot of AI apps basically make the experience easier by giving u a nice UI, hosting, and extra features around an AI model. So if u have decent hardware and don't mind setting things up yourself, local AI can save u a lot of money. There r many places where u can get these models, like Ollama, Hugging Face, and many more. But I personally like Hugging Face because u can actually find a huge variety of AI models there, from small models for potato PCs to some seriously powerful ones. So, to get these models, head over to the official Hugging Face website. Now, go to the Models section and select Trending in the sort option, same as shown in Fig. 10,11
As we go through the website, there r multiple types of models, like cloud AI, local AI models, image models, etc. But what we want right now is a local LLM. For that, we're looking for a GGUF file. So head over to the model search — not the top search bar, but the search bar near the Models section — and search for GGUF. Now u will specifically see models available in the GGUF format. There r multiple models like Qwen, Gemma, Kimi, DeepSeek, and many more, with sizes ranging from a few million parameters to billions and even larger. (Those who don't know what parameters are can click on the blue term and get the meaning in more depth.) Now, my machine can comfortably run around 2B–5B parameter models (where B means billion parameters), but I even struggle with some 9B models. T-T
I have a fix for that too: quantization. U can choose different quantized versions to reduce memory usage and improve performance. I generally use 4-bit models, but u can go lower, like 2-bit, if ur system is weaker. If u have a stronger system, u can also go for higher-quality quantizations. So now, find any model of ur choice, click on it, and download it. I actually recommend downloading and testing multiple different models and sizes so u can understand how much load ur machine can handle. Also, when u click on a quantized model, u will usually see useful details about it, like the context length (basically how much information the model can consider during a conversation) and other technical stuff. U can also check the model details and recommendations to figure out which quantized version suits ur system best by login into the site. :D Refer to Fig. 12 and Fig. 13 for clarity.
For my cybersecurity bros, yes, u can also experiment with models that are less restrictive than mainstream AI assistants. Since these models can be run locally, u get much more control over the model, system prompts, and overall setup. But obviously, use that freedom responsibly — local AI gives u more control, not a free pass to do stupid illegal stuff T-T. Just write &other=uncensored in the URL after sort=trending, as shown in Fig. 14. Now, these models can be really useful for cybersecurity because they can help u understand how attacks work, how systems can be protected, and they may be less restrictive than mainstream LLMs. These models can be useful for cybersecurity research and learning because they may be less restrictive when discussing technical security topics, helping u understand how attacks work and, more importantly, how to defend against them.
Now, after downloading the models, you can rename them however you like, such as xyz.gguf or abc.gguf, and then move them into the models folder on our flash drive. Make sure you have a high-storage flash drive like 32GB. And honestly, 64GB is actually overkill for now—don’t go for that unless you have a Godzilla-type PC that can handle 32B models too :)
Here, we now have our shell, engine, and wheels for our car. What we need now is fuel—the thing that starts our engine. Now, since you know I’m lazy xD, for now we’re just going to make a simple .bat executable file. But in the later stages, we will create our own launcher too. So here’s the simple analogy: the .bat file will communicate between the engine and the models folder and help us run our AI locally. So, to do this, create a new .txt file inside the pendrive, but not inside any folder. Open it and take a screenshot of the code and give it to ChatGPT with the prompt: “Extract the code from this image”, then copy and paste the extracted code. After that, change the model names in the code according to the names you gave to your .gguf files. They should match exactly with the filenames inside your models folder. You can refer to Fig. 15 and Fig. 16 for better clarity :D
After that, hit save, but remember to save it as a .bat file (name it run.bat or start.bat, whatever you wish :D). And boom—you just conquered the hottest achievement in my blog: “The Doom Scroller”. Thanks for making it all the way to the end, brotha :D. So now, to run your models, simply click on the newly created .bat file. It will ask you which model you want to boot. Press the number corresponding to your model and hit Enter. It will then redirect you to a webpage, but you might see an error saying that the server has not started yet, as shown in Fig. 17. Don’t panic T-T. Most likely, your model is still loading. Just refresh the webpage after a while and check the terminal from your taskbar. Once you see something like:
listening on http://127.0.0.1:8080 as shown in Fig. 18
your AI is officially ready to go :D. Generally, when you start a model for the first time, it can take a few minutes to load depending on your model size and your PC. Once it's ready... enjoy, man. Ask whatever you want, experiment with different models, test their capabilities, break things, fix things, and basically have fun with your own portable offline AI, brotha ;)
Now, I used it and asked it a classic philosophical question: "The Ship of Theseus". I asked it, if we have a ship supposedly named “Bob”, and we slowly replace all of its body parts until it has no original parts left, then collect all the oldoriginal parts and arrange them together to make another ship called Bob... which one would actually be the real Bob? Or would both of them be Bob?
You can see the model’s response in Fig. 19 and Fig. 20.
Also, if the model you downloaded supports image understanding, you can give it images too and ask questions about them. Some models can also work with documents and other files, depending on the interface and model you are using.
But the objective here isn’t to create our own pocket version of ChatGPT or Claude. The real goal is to create a portable, flexible, and convenient AI system where our data stays on our own device instead of being sent to some huge MNC’s servers. You can travel with your pendrive, plug it into a compatible machine, and run your AI directly from it, because your models and AI setup are stored on the flash drive while the AI uses the hardware of the machine you plug it into. So yeah... your AI travels with you, but the computing power comes from wherever you plug it in. Pretty cool, right? :D
Now, to turn it off, simply closing the webpage isn’t enough. Head over to ur terminal, press Ctrl + C, and then press Y followed by Enter if it asks for confirmation. This will automatically terminate the running session, same as shown in Fig. 21.
We created our own portable Local AI setup (yeah, we did take a shortcut xD).
Comments
Post a Comment