Why to run AI models locally?⌗

Well, if you are there, you might probably know the answer of this question, but just for sake of intro, Running models locally provides the following benefits:

  • Cost reduction
  • Unlimited usage
  • More privacy
  • More control

So, recently i have been trying to find an model that can be actually usable on my specific hardware. Since, the beginning of the local AI models, i have tried many of those, either they were too big in size, or needs me heavy GPU, which ofcourse was not my case.

I started exploring different small models then, like smollm, tinyllama, which were good, but they doesn’t provide the tool calling feature, which kind of make them useless (in my case).

What is tool calling anyway?⌗

The tool calling means, the AI can read,write,execute for you using the tools provided by you agent or any program. Before we dive in, i want you to tell that, AI can’t actually execute anything or make actually an action. I can only work with text’s input/output. So how does it even work?

There’s come the JSON in the picture, AI’s use the JSON structure to create an request of tool call with all the arguments and information, then the agent you are using handles the actual execution by extracting that json and making it execute exactly.

My hardware⌗

I have 16GB 2400mz ram, 1TB HDD, CPU: Intel(R) Core(TM) i3-9100F CPU @ 3.60GHz, No usable gpu.

Models i tried.

I tried running tinylllama:1.1b model of size 637 MB. it gaves 23 token/s. which is i think ok ok to use locally, but it doesn’t support tooling and that’s where it feels useless to me.

❯ ollama run tinyllama --verbose
>>> hello.
Hi there! I'm glad to hear that you found my comment useful, and I'd love to help with your request. Please let me know what questions or concerns you have about the article. Whether
you're interested in learning more about a specific topic or just looking for general information, feel free to ask away. I'll do my best to provide you with helpful answers! Don't
hesitate to reach out if you need any further assistance. Best regards,
[Your Name]

total duration:       4.673213147s
load duration:        29.595312ms
prompt eval count:    36 token(s)
prompt eval duration: 118.675088ms
prompt eval rate:     303.35 tokens/s
eval count:           103 token(s)
eval duration:        4.47597143s
eval rate:            23.01 tokens/s

I even tried smollm of 1.7b of size 990 MB something, and here’s its token evaluation:

total duration:       1m19.192179911s
load duration:        30.771139ms
prompt eval count:    179 token(s)
prompt eval duration: 384.048969ms
prompt eval rate:     466.09 tokens/s
eval count:           972 token(s)
eval duration:        1m18.105162196s
eval rate:            12.44 tokens/s

same, it doesn’t support tooling, not useful.

❯ ollama run qwen2.5:1.5b --verbose
>>> hello
Hello! How can I help you today? Is there anything on your mind that you
would like to talk about? I'm here to listen and answer any questions you
might have.

total duration:       3.256996963s
load duration:        91.701436ms
prompt eval count:    30 token(s)
prompt eval duration: 428.660945ms
prompt eval rate:     69.99 tokens/s
eval count:           37 token(s)
eval duration:        2.662413615s
eval rate:            13.90 tokens/s

❯ ollama run smollm:135m --verbose
>>> what can you do?
What type of job is this, and what kind of person would be interested in it?

total duration:       483.184523ms
load duration:        32.524471ms
prompt eval count:    56 token(s)
prompt eval duration: 23.91895ms
prompt eval rate:     2341.24 tokens/s
eval count:           19 token(s)
eval duration:        413.605136ms
eval rate:            45.94 tokens/s

GGUF⌗

Then i tried gguf

./build/bin/llama-cli \
  -m ~/models/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf \
  -t 4 \
  -c 2048 \
  --temp 0.7

Running with llama.cpp and guff is actuall fast. the qwen2.5:1.5b is actually working, but sometimes fails to call tool efficiently. and a bit dumb.