Summary
I’ve always enjoyed giving old hardware a second life by turning it into server nodes, and the steady rise in PC hardware prices has only reinforced that habit. As LLMs have become increasingly useful for everyday productivity, I’ve found myself regularly repurposing aging systems into dedicated AI workstations. With the right model and a few optimizations, even older hardware can deliver surprisingly capable LLM-hosting performance.
I’m not referring to enterprise-grade servers, either. My setup revolves around a 2017 Acer Predator Helios 300 (G3-571-77QK) equipped with a GTX 1060 (6GB) and 16GB of DDR4 RAM. Although it’s powered by an aging Pascal-era GPU, the laptop performs exceptionally well as a llama.cpp server while also handling the rest of my containerized productivity stack.
Why I Chose Proxmox for My Local LLM Server
Although lightweight Linux distributions such as DietPi, AntiX, Fedora, and Linux Mint are all excellent choices for a project like this, I decided to stick with Proxmox. I’d already installed it on the laptop months earlier as part of another experiment, and its powerful container management and isolation capabilities made it the ideal platform for replacing ChatGPT and other cloud-based AI services with a completely local environment.
After hearing reports of GPU passthrough issues on newer versions of Proxmox with GTX 10-series graphics cards, I chose to remain on Proxmox 9.0. Likewise, because Nvidia has dropped support for Pascal GPUs in its latest driver releases, I stayed with version 580.
Once the Nvidia drivers were installed on the host system, I configured GPU passthrough for the LXC container by modifying its configuration files and assigning the required permissions. Inside the container, I repeated the driver installation process, this time using the --no-kernel-module option.
The overall procedure closely mirrored the setup I previously used for a GTX 1080-powered llama.cpp server running on an older gaming PC. The only significant difference involved selecting the appropriate drivers for the laptop’s Pascal GPU.
Configuring llama.cpp Inside an LXC Container
While GPU passthrough itself wasn’t particularly difficult, getting llama.cpp to work correctly with the graphics card required considerably more effort.
Based on my previous experience, I opted to build a Vulkan-enabled version of llama.cpp. I first installed the necessary Vulkan packages on the host using:
apt install libvulkan1 vulkan-tools
Afterward, I verified that the GPU was correctly recognized by running:
vulkaninfo | grep -i deviceName
Inside the LXC container, I installed all of llama.cpp‘s required dependencies, including Vulkan libraries, shader tools, build packages, Git, CMake, Curl, and related development utilities.
To ensure Vulkan functioned properly inside the container, I edited the Nvidia Vulkan configuration file located at:
/usr/share/vulkan/icd.d/nvidia_icd.json
and added the required configuration with the proper formatting.
Running:
vulkaninfo | grep -i deviceName
confirmed that the GTX 1060 was available as a Vulkan device.
Next, I cloned the llama.cpp repository using Git, entered the project directory, and compiled it from source with Vulkan support enabled using CMake.
Gemma 4 E4B Is a Great Match for a GTX 1060
With 6GB of VRAM and 16GB of system memory, my laptop could comfortably run several compact language models. Since I wanted a balance between responsiveness and accuracy, I ultimately selected Gemma 4 E4B, avoiding the compromises that often come with models smaller than 2B parameters.
Because I already had the model’s GGUF files stored on another computer, I transferred them directly to the llama.cpp container using SCP rather than downloading them again.
From the compiled /build/bin directory, I launched the model using llama-server, loading both the primary GGUF model and the accompanying mmproj file.
Several launch options were especially important:
--ngl 999forced inference to run on the GPU.--mmprojenabled multimodal vision support.--webui-mcp-proxyallowed integration with MCP servers.
Performance Exceeded My Expectations
The results were surprisingly impressive.
The system consistently generated more than 30 tokens per second. While that’s roughly one-third the speed of my RTX 3080 Ti, it’s still more than fast enough for practical everyday use.
Even when paired with Home Assistant, the setup responds quickly enough to handle real-time interactions.
I’ve also tested Gemma 4 E4B across a variety of workloads, including:
- Image recognition
- Document summarization
- RAG workflows
- Semantic search using nomic-embed-text for embeddings
Across each of these tasks, the model has delivered consistently solid performance.
Why I Replaced ChatGPT With a Self-Hosted LLM
My decision to replace ChatGPT with a locally hosted model ultimately comes down to privacy.
Although ChatGPT offers free access and convenient cloud-based AI services, I’d rather avoid sending sensitive personal data to external servers for inference.
One example is my Paperless-ngx workflow, where Paperless-GPT handles OCR processing and Paperless AI provides contextual conversations and automatic document tagging. Uploading documents such as bank statements, identification cards, or academic records to a cloud-hosted AI service raises privacy concerns I’d prefer to avoid.
The same reasoning extends to other tools in my workflow, including Open Notebook for research, Karakeep for bookmark management, Open WebUI as a voice assistant, and Blinko for note-taking. By keeping inference entirely local, I retain greater control over both my data and my productivity environment.