I Replaced ChatGPT with Local AI: Running Qwen 3.5 and Multimodal Models for Free Using Ollama

TL;DR: You can run Qwen 3.5 (7B) locally at 45+ tokens/sec on an RTX 3060, process images with vision models, and add voice commands using Whisper + Piper TTS. Total cost: $0/month after hardware. Setup takes 20 minutes.

The Problem: API Costs and Data Lock-in

I was spending €23/month on ChatGPT Plus. Not bad, until I needed to process 200 product descriptions for an e-commerce client. The API bill hit €47 in one afternoon. Worse: I couldn’t verify if my proprietary prompts were training OpenAI’s next model.

For businesses handling client data or healthcare records, this is a liability, not just an expense. Local AI solves both. You pay for hardware once (or use existing equipment), and your data never leaves your machine.

The Solution: Ollama + Efficient Small Models

Ollama isn’t just a wrapper around llama.cpp. It handles model quantization, GPU offloading, and provides a clean API that replaces OpenAI’s endpoints. Combined with modern 3B-14B parameter models, you get 90% of GPT-4’s reasoning at 5% of the operational cost.

Why Qwen 3.5 and 3.6?

Alibaba’s Qwen series dominates local AI right now because:

• •Apache 2.0 license — Commercial use allowed without legal gray areas
• •Multilingual — Handles Romanian, Chinese, and English better than Llama-3 in my tests
• •Efficient architecture — The 7B model outperforms Llama-3 8B on reasoning benchmarks while using less VRAM

The 3.5 and 3.6 releases (late 2025/early 2026) fixed the “instruction following” issues from Qwen2. They now reliably output JSON and follow system prompts without drifting.

Hardware Requirements (Real Numbers)

Before you download anything, check your specs:

Minimum (CPU Only)

• •RAM: 16GB DDR4
• •Storage: 20GB free SSD space
• •Speed: 4-8 tokens/sec (usable for testing, painful for production)
• •Models: Qwen 3.5 3B, Phi-4 3.8B

Recommended (GPU Accelerated)

• •GPU: NVIDIA GTX 1070 8GB or RTX 3060 12GB
• •RAM: 32GB DDR4 (allows OS + browser + model context)
• •VRAM: 8GB+ for 7B models at Q4 quantization
• •Speed: 35-60 tokens/sec
• •Models: Qwen 3.5 7B/14B, Llama 3.2 Vision 11B

Optimal (Multimodal + Voice)

• •GPU: RTX 4070 Ti Super (16GB) or Apple Silicon M3 Pro (18GB unified)
• •RAM: 32-64GB
• •Speed: 60-120 tokens/sec, simultaneous TTS/STT processing
• •Models: Qwen 3.6 32B (Q4), Llama 3.2 Vision, plus Whisper Large V3

Note: AMD GPUs work via ROCm, but NVIDIA has better Ollama support currently. Apple Silicon M-series chips are surprisingly competitive for 7B-13B models thanks to unified memory.

Setup: From Zero to Local ChatGPT

Step 1: Install Ollama

1# macOS/Linux
2curl -fsSL https://ollama.com/install.sh | sh
3
4# Windows
5# Download installer from ollama.com/download

Verify installation:

1ollama --version
2# ollama version 0.6.3 (or newer)

Step 2: Pull Your First Model

Start with Qwen 3.5 7B — the sweet spot for quality vs. speed:

1ollama pull qwen3.5:7b

The :7b tag defaults to Q4_K_M quantization (4-bit). This reduces model size from ~14GB to ~4.5GB with minimal quality loss.

Step 3: Test Performance

Run a benchmark to check your tokens/sec:

1ollama run qwen3.5:7b
2>>> Why is the Romanian tech sector growing faster than neighboring countries?

Time the response. On an RTX 3060 12GB, you should see ~45 tokens/sec. On an M3 MacBook Pro, ~55 tokens/sec.

If you’re getting <15 tokens/sec, your model is running on CPU. Check GPU detection:

1ollama ps
2# Should show GPU % usage, not just CPU

Multimodal: Processing Images Locally

Qwen 3.5 Vision and Llama 3.2 Vision can analyze screenshots, charts, and documents. Here’s how to use it:

1import ollama
2
3response = ollama.chat(
4    model='qwen3.5-vl:7b',
5    messages=[{
6        'role': 'user',
7        'content': 'Extract all prices from this receipt image',
8        'images': ['receipt.jpg']
9    }]
10)
11
12print(response['message']['content'])

Performance note: Vision models require 2-4GB additional VRAM. The 7B vision model uses ~6.5GB VRAM total. If you only have 8GB VRAM, close Chrome tabs before running.

Adding Voice: STT and TTS Integration

Ollama is text-only. For voice, you need two additional pieces:

Speech-to-Text (Whisper)

Use faster-whisper (CTranslate2 optimized) for local transcription:

1pip install faster-whisper

1from faster_whisper import WhisperModel
2
3model = WhisperModel("base", device="cuda", compute_type="float16")
4segments, info = model.transcribe("audio.mp3", language="ro")
5text = " ".join([segment.text for segment in segments])
6
7# Send to Ollama
8response = ollama.generate(model='qwen3.5:7b', prompt=text)

Base model uses ~1GB VRAM and processes Romanian at 50x real-time speed on GPU.

Text-to-Speech (Piper)

Piper is a fast, local neural TTS that runs on CPU:

1# Install piper
2pip install piper-tts
3
4# Download Romanian voice (if available) or use English
5# Voices from https://huggingface.co/rhasspy/piper-voices/tree/main

1import piper
2
3voice = piper.Voice(model_path="ro_RO-gabriela-medium.onnx")
4audio = voice.synthesize(response['response'])
5# Play audio or save to file

Full Pipeline Architecture:

1Microphone → Whisper (STT) → Ollama (LLM) → Piper (TTS) → Speaker

Latency breakdown on RTX 3060:

• •STT: 200ms (3-second utterance)
• •LLM (50 tokens): 1.1 seconds
• •TTS: 300ms
• •Total: ~1.6 seconds (comparable to cloud APIs)

Results: What I Actually Measured

I tested Qwen 3.5 7B against GPT-3.5-turbo on three business tasks:

Speed benchmarks (RTX 3060 12GB):

• •Qwen 3.5 7B Q4: 47 tokens/sec
• •Llama 3.2 8B Q4: 42 tokens/sec
• •Phi-4 14B Q4: 28 tokens/sec (slower but smarter)
• •Qwen 3.5 Vision 7B: 38 tokens/sec (image + text)

Trade-offs and Limitations

You lose:

• •Internet knowledge cutoff — Local models don’t browse the web (unless you add RAG/search APIs)
• •Huge context windows — Running 128K context locally requires 32GB+ VRAM. Most local setups use 4K-8K context
• •Zero-setup — Initial tuning takes 2-3 hours. Ollama makes it easier, but it’s not “sign up and chat”
• •Best-in-class reasoning — GPT-4 and Claude 3.5 Sonnet still beat 7B local models on complex logic. You need 30B+ models to match them, requiring expensive hardware

You gain:

• •Zero latency variance — No “server overloaded” errors
• •Privacy by default — HIPAA/GDPR compliance without vendor audits
• •Customization — Fine-tune on your data with tools like unsloth

Conclusion

I canceled ChatGPT Plus after three weeks of local testing. For 80% of my tasks — drafting emails, analyzing PDFs, writing code scaffolding — Qwen 3.5 7B matches GPT-3.5 quality at infinite scale and zero marginal cost.

Start with Ollama + Qwen 3.5 7B on your existing hardware. If you get 30+ tokens/sec, you’re good to replace your daily driver. If not, a used RTX 3060 (€200 on Romanian OLX) pays for itself in 10 months of ChatGPT Plus savings.

Next steps:

1. Install Ollama and test Qwen 3.5 7B
2. Set up a document folder for RAG (use anythingllm.com for the UI)
3. Integrate Whisper for voice commands

Your data stays yours. Your wallet stays full.

Further Reading

• •Ollama Model Library — Official supported models and quantization tags
• •Qwen Technical Report — Architecture details and benchmark methodology
• •LocalLLaMA Subreddit — Community hardware benchmarks and optimization tips
• •Piper TTS Voices — Multilingual voice models including Romanian
• •Faster-Whisper Documentation — Local speech recognition setup guide