Run a local LLM with Ollama

1. Install Ollama
2. Pull and run your first model
3. Talk to it over HTTP
4. Run it on a server
5. Try other models

Ollama packages an open large language model, its weights, and a small HTTP server into a single command. You download a model once and run it entirely on your own hardware, which means no API keys, no rate limits, and no data leaving your machine. It's the fastest path from "I want to try an LLM" to a working prompt.

#Install Ollama

On Linux the one-line installer sets up the binary and a systemd service:

curl -fsSL https://ollama.com/install.sh | sh

On macOS download the app from the website, or use Homebrew:

brew install ollama

The installer registers Ollama as a background service. Confirm it's running:

ollama --version
systemctl status ollama   # Linux

#Pull and run your first model

Pick a model and run it. Ollama downloads the weights on first use, then drops you into an interactive prompt:

ollama run llama3.2

The first run pulls a few gigabytes, so give it a moment. After that the model is cached and starts instantly. Type a question, get an answer, and press Ctrl+D to exit.

If your machine is modest, start small. A 1B model is fast even on a laptop:

ollama run llama3.2:1b

To see what you've downloaded and reclaim space:

ollama list
ollama rm llama3.2:1b

#Talk to it over HTTP

The reason Ollama is useful in real applications is the built-in REST API on port 11434. Anything that speaks HTTP can use it:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain a reverse proxy in one sentence.",
  "stream": false
}'

It also exposes an OpenAI-compatible endpoint at /v1/chat/completions, so most libraries written for OpenAI work against Ollama by just changing the base URL. That makes it a drop-in local stand-in while you develop, before you decide whether to use a hosted API like OpenAI in production.

#Run it on a server

Running Ollama on your laptop is fine for experiments, but for a shared model you'll want it on a server with a GPU. The principle is the same, with two changes: bind the API to all interfaces and put it behind a reverse proxy.

sudo systemctl edit ollama

Add the host override:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Do not open port 11434 to the internet directly, it has no authentication. Front it with Nginx and require a token, the same way you'd protect any internal service. For a full walkthrough of sizing a box and locking it down, see self-host an LLM on your own server.

#Try other models

Ollama's library covers most popular open-weight models. A few worth knowing:

• llama3.2 — a solid general-purpose default from Meta.
• deepseek-r1 — a reasoning model, see run DeepSeek locally.
• qwen2.5-coder — tuned for code generation.
• nomic-embed-text — an embedding model for building search and retrieval.

Swap the name in ollama run or ollama pull to try any of them. Because everything runs locally, the only limit is your hardware.