Algeria AgencyInfrastructure
Running a model on your own machine: the three numbers that decide
The number in a model’s name does not say whether it will fit. Three others do, and they are worked out before buying anything.
You are offered a machine, and the quote talks about billions of parameters, quantisation and a context window. Those three words decide whether the installation will work, and they almost never appear in the same sentence as the price.
They are nonetheless three numbers, they multiply together, and the calculation fits on one line. Somebody who has never opened a machine can do it in five minutes and know whether the proposal is serious before committing to anything.
This article gives the three numbers, the calculation, and what happens when the result does not fit — because nothing stops, everything slows, and a slowdown looks like nobody’s fault. It does not cover the choice between hosting and calling, which is covered in full elsewhere, nor what the model will do once installed.
The number in the name does not decide
Models carry a number: seven, thirteen, seventy. Those are billions of parameters, and it looks like a size. People naturally conclude that a big machine takes a big model, and that conclusion is right for a different reason from the one imagined.
A parameter is a number to be stored. What decides is therefore not the quantity of parameters in itself, but the space each one occupies — and that space is chosen. The same seven-billion model can demand fourteen gigabytes or four, according to a setting whose name never appears on the cover.
That is why a salesperson can be telling the truth in saying their machine runs a given model, and the installation still be disappointing. It runs it in a reduced version, which is perfectly legitimate and perfectly invisible.
So the question to ask is not "which model" but "which model, in which format, with which window". Three numbers. The rest of this article takes them one at a time.
First number: the card’s memory
The computation runs on a graphics card, and the card has its own memory, separate from the machine’s. That figure — eight, sixteen, twenty-four, forty-eight gigabytes — is the ceiling. The machine’s memory does not replace it and does not usefully add to it.
It is the number most often missing from a quote, and its absence is not always a manoeuvre: many commercial sheets lead with computing power, which is a more flattering figure and decides nothing while the model does not fit.
It is asked for in one line, and the answer is a number, not a range. "A recent professional card" is not an answer; "twenty-four gigabytes of video memory" is one.
One trap lies in how cards are named. A single commercial name often covers two or three memory quantities, sold as the same product and distinguished by a suffix nobody says aloud. Two quotes can therefore announce the same card and offer two different ceilings, which makes the comparison wrong at its most important line.
Everything that follows is compared against that ceiling. If you were to keep one figure from a quote for a machine intended for a model, it would be that one.
Second number: precision, and what it costs
Each parameter is stored in a format, and the format is counted in bits. Sixteen bits are two bytes, which gives the basic rule: a seven-billion-parameter model at sixteen bits occupies about fourteen gigabytes. That is a definition, not a measurement — two bytes per parameter, multiplied by the number of parameters.
Quantisation means reducing that format: eight bits, six, four. At four bits the same model comes down towards four gigabytes and fits on a consumer card. It is the manoeuvre that makes a local installation possible, and it is used almost everywhere.
It is not free. Reducing precision degrades answer quality, slightly at first and clearly below a certain point, and the degradation does not show up as outright errors: it shows up as answers that are a little flatter, a little less accurate on hard cases, and on less well-represented languages — Arabic among them.
The degradation is checked without hardware: take ten of your hardest questions, run them through both formats at the supplier, and read the twenty answers side by side. If you see no difference on your own cases, the reduced format is the right choice and it saves you a card.
None of that is an argument against quantisation, which is an excellent compromise. It is an argument for the format being written into the quote, and for two proposals being compared at equal format.
Third number: the window consumes too
The context window is how much text the model can have in front of it at once: the question, the documents attached to it, and the whole conversation history. It is counted in tokens, a token being roughly three quarters of a word in French and less in Arabic.
What is systematically forgotten is that it occupies memory, on the same card, on top of the model. A large window open for several simultaneous conversations can consume as much as the model itself.
That is why an installation that works perfectly in a demonstration becomes unstable in service: the demonstration is one short conversation, and real use is ten long ones.
The consequence to keep is that the window is a purchase, not a property. Doubling an installation’s advertised window costs memory, so it costs card, so it costs money, and that has to appear somewhere.
The calculation, on one line
Parameters multiplied by bytes per parameter, plus a margin for the window and for running. A seven-billion model at four bits is about four gigabytes; add half again for the window and the rest, and you are around six. An eight-gigabyte card is enough; a six-gigabyte card is not.
The same model at sixteen bits demands fourteen, plus the margin: twenty-one. A twenty-four-gigabyte card passes, a sixteen does not. That is the whole arithmetic, and it is redone with your figures in two minutes.
The margin is not a magic number: it covers the model server’s own working memory and the window, and it depends on how many conversations you want to serve at once. A single user fits in a small margin; twenty simultaneous calls do not.
That calculation decides the purchase and decides nothing else. The monthly running cost — electricity, occupancy, the person watching — is reasoned separately, and this site’s calculator handles the called side rather than the hosted one.
What happens when it does not fit
Nothing stops. That is the part to know before buying, because it explains most disappointments: when the model exceeds the card’s memory, the system moves the surplus into the machine’s memory, and it works.
It works ten to fifty times more slowly. An answer that took two seconds takes a minute, and nothing in the system reports an error, because there is none: the computation happens, elsewhere, more slowly.
The symptom is therefore "it is slow", which is the hardest complaint to diagnose because it has twenty possible causes. It gets blamed on the network, the model, the supplier, the machine in general — and the cause is an overrun of a few gigabytes that a two-minute calculation would have announced.
Confirming it takes ten minutes and needs no particular skill: you open the card’s monitoring tool, ask a question, and watch whether the video memory fills to the ceiling while the answer is being produced. If it sticks there, that is the cause. It is a gesture whoever installed the machine can show you once, after which you will know how to repeat it.
It is also why a successful demonstration proves nothing about the delivered configuration. Ask which card the demonstration runs on, and in which format. Two questions, and they separate an honest demonstration from one performed elsewhere.
Fitting is not answering fast
Once the model is in memory, a second question arises and it is independent of the first: how many people can talk to it at once. A card that holds the model serves one user comfortably and degrades as you add more.
It is not a linear, polite degradation. It is flat up to a point, then the queue lengthens and waiting times climb fast. That point depends on the length of the answers and on the window open per conversation, so on how your people work.
The practical consequence is that an installation is sized on concurrency, not on headcount. Twenty people asking three questions a day almost never overlap; five people handling a continuous flow overlap constantly.
There is a consequence discovered in service rather than in the quote: under queue it is not only the waiting that rises, it is the memory too, because each conversation in progress keeps its own window open. A machine that held ten conversations holds six on the day answers get longer, and nothing on the card has changed.
That figure is not calculated on a whiteboard: it is measured over two weeks of real use, noting the hours. It is the same log the hosting decision calls for, and it serves twice.
The machine is not only the card
Around the card, four lines decide whether the installation lives: the machine’s memory, which must at least equal the card’s to load the model; the disk, because a model weighs as much on disk as in memory and you keep several; the power supply; and the cooling.
The last two are infrastructure subjects rather than computing ones, and they are underestimated in a hot climate. A card that gets hot reduces its own speed to protect itself, without reporting anything, and the symptom is again "it is slow" — the same symptom as the memory overrun, from an opposite cause.
The power supply is where the question of the mains meets this one. A machine of this type draws more than an ordinary server, at peak, and the circuit feeding it is rarely the one you would have chosen.
There remains where it is put, and that is rarely discussed before delivery. A machine of this type is noisy under load, more so than an office server, and it warms the room it sits in. Installed in a shared office, it ends up in a corridor or in a room nobody air-conditioned, which brings the heat question back through the service door.
None of those four lines makes anybody dream and all four appear after delivery if they do not appear before. They are asked of the supplier at the same time as the card’s memory.
What "on our premises" does not guarantee
A machine on your premises settles a real question: your texts do not go to a third party. That is a sufficient motive on its own for certain files, and it is the only motive we accept without argument.
It does not settle updating, backup, or access. A server under a desk, reached through a shared account, unpatched and with no copy elsewhere, is less protected than a remote service run by people whose job it is. Location is not a security measure, it is an answer to a question of destination.
Nor does it say anything about who at your end looks after it. A local model needs an owner, like any installation, and the exit delay and the second copy remain exactly the same questions as before.
The word circulating in proposals is "sovereign", and it has no content until you say sovereign over what. Sovereign over the data, over the model, over the machine, over the patches: those are four things, they are bought separately, and a machine in a corridor gives one of them.
What the quote does not cost
Three expenses accompany the purchase of a machine of this type and are rarely in the same column as its price. We give no amount for any of them here, and the reason is written below.
The first is shipping and duty. The amount depends on the tariff heading applied and on the regime in force at the time of the operation, which is checked with a freight agent for your precise equipment. A figure published in an article would age badly and be quoted long after it stopped being true, which is exactly the defect this blog forbids itself.
The second is the warranty. Equipment of this type is often warranted from abroad, and a warranty that assumes a return is a warranty whose delay is a journey. Ask who performs the replacement and where the part is.
A fourth is added if the machine runs a licensed operating system, which is common when it joins an existing estate: what you actually buy when you buy a server is not the same list as what you buy when you buy a card, and the two invoices do not arrive together.
The third is the spare-part delay, and it is the one that decides continuity. A card that fails while the service is running on it is not a hardware incident: it is a stopped service, for the length of the delay, and that delay is asked about before rather than after.
The check: four lines to ask for
Write these four requests in an email and send them to the supplier. First: which card, and how many gigabytes of video memory. A number, not a range.
Second: which model exactly, and in which quantisation format. Second number. Third: which context window, and for how many simultaneous conversations. Third number.
Fourth: who replaces the card if it fails, where the part is, and within what delay. That question is not technical and it is the only one of the four whose answer commits somebody.
With those four answers, section 5’s calculation takes two minutes and you know whether the proposal holds. Without them, two quotes are not comparable, whatever the demonstrations show and whatever the price gap.
What we do, and what we refuse
We put the four questions on your behalf when you ask us to, we redo the arithmetic on the proposed configuration, and we tell you whether it fits — including when the answer is that the need does not justify a machine at all.
We measure real concurrency over two weeks before sizing anything, because it is the only figure in this file that cannot be deduced and it decides half the expenditure.
We refuse to publish a duty amount or a machine price. Those figures move, they depend on your equipment and your operation, and a figure published here would be quoted by somebody six months after it stopped being true.
We also refuse to sell a local installation as a security measure. It answers a question about where the data goes and no other, and what the machine will run stays a decision separate from buying it. What you can do without us is section 11’s email: four lines, and two quotes become comparable.
Frequently asked questions
Is a powerful laptop enough?
For trying things out, often yes. For serving several people, no, and the reason is not power but video memory and cooling: a portable machine reduces its speed to stay cool, and it does so precisely when the load lasts.
Can a second card be added later?
Sometimes, and it is checked before buying rather than at the moment it is wanted. The power supply, the physical space and the motherboard decide, and a machine chosen without that question gets replaced instead of extended.
Does quantisation damage answers much?
Little up to a point, clearly below it. What matters for you is comparing two proposals at the same format, and testing on your own hard cases rather than on general questions, because that is where the difference appears.
Does a local model need an internet connection?
Not to answer, which is the point of it. It needs one to install, to update and to monitor remotely, and a machine cut off from the network requires somebody to travel for each of those three things.
How do we know whether our current machine can do it?
Look at the card’s video memory, apply section 5’s calculation, and compare. It is a five-minute operation needing no tools and it answers in most cases.
Does a local model cost less than a service?
At low usage, no, because the machine costs on the days with no traffic. The reasoning is done on occupancy rather than on volume, and it is handled separately — the hosting decision and the buying decision are not taken in the same order.
Where we come in
A card named without its gigabyte figure is the one line of the quote you are actually missing.
- We price the memory the exact offered model needs, in the exact version it is offered in.
- The concurrency log runs for two weeks at your end before any hardware is ordered.
- The spare part’s location and its delay come out of the supplier in writing.
No customs figure will come from here: it depends on your equipment’s tariff heading, which only a freight agent establishes.
Read next
Infrastructure: where your data lives, and how long it takes to leave
The only family in this pillar that produces nothing visible. The useful question is not local or cloud: it is the notice period for getting out.The mains before the UPS: earth, circuit and heat
A UPS fixes neither a missing earth, nor a shared circuit, nor the heat that follows an outage. What damages equipment upstream of it, and how to check.Cloud hosting: what costs is not the storage
The line everybody compares is the cheapest on the invoice. The other three get paid on the day you need them most.
Let us talk about your project
A free audit, no commitment: we look at your online presence and tell you what is holding it back.