Used Nvidia V100 GPUs Run DeepSeek and Llama Models at Home

Used Nvidia GPUs are proving capable of running sizable AI models at home. One builder says a $9,450 server with 12 secondhand V100s runs DeepSeek V4.1 Flash experts in 4-bit NVFP4, producing its first code token in 58 seconds on a 256K context—down from 108.4 seconds after an earlier upgrade. Prompt reading exceeded 4,500 tokens per second beyond 32K, while three concurrent requests reached a combined 360 tokens per second for code and 213 for legal tasks. The tests measure speed, not answer quality.

A separate $1,042 system with three used V100s ran Qwen 3.6 35B at 100.8 tokens per second with a 4K context and 72.7 at 128K. It was 19% faster than an M5 Max Mac Studio at 128K, though the systems used different software; two V100s reached 77.1 tokens per second at that context length. Practical model choices also depend on memory: 16 GB of RAM can run 4-bit Qwen 3 8B and Llama 3.1 8B, while 8 GB supports smaller options such as Llama 3.2 3B, Qwen 3 4B and quantized Mistral 7B.

Security concerns run alongside AI adoption. A GAO report says AI can make phishing and deepfakes more convincing and compress reconnaissance from weeks to hours or less; autonomous agents may carry out multiple attack stages with little human involvement. The FBI’s Internet Crime Complaint Center recorded more than $20 billion in losses across over 1 million complaints in 2025, including more than $893 million tied to AI-related cybercrime. The report also flags prompt injection, weak testing of AI security tools, fragmented rules, unpredictable costs and a shortage of trained workers. AI can help defenders, but the report says attackers currently benefit from its speed.

Organizations are still working out how much authority to give agents. A survey found 55% of security leaders fear AI-driven attacks will outrun their teams, while only 1% are comfortable letting agents handle most tasks without approval; 7% expect to feel that way within a year. Former FBI investigator Paul Bingham warns that companies may give agents credentials before limiting access or learning their normal behavior. A VB Pulse survey found 46% of enterprises had an agent incident or near miss in the past year. Among organizations giving agents distinct identities, 60% said they could stop a rogue agent within minutes, compared with 33% of those using shared keys. Bingham also cautions against letting agents approve multi-factor authentication prompts.

A Microsoft outage on Nov. 12, attributed to a software update problem rather than a cyberattack, grounded planes and disrupted banking systems. The article uses the incident to illustrate the dependence of everyday services on technology and warns about malicious uses of AI, including autonomous AI hackers. Separately, a Marsh survey of 6,700 employees in 10 countries found AI training reaches 78% of executives but only 32% of entry-level professionals and 20% of hourly workers. Regular AI use also varies: 64% of executives use it regularly, versus 20% of entry-level respondents. Overall, 37% use AI regularly and 71% have tried it at least once; Marsh says the training divide may contribute to stalled adoption.

In travel, Booking.com’s B2B arm is positioning itself as an AI resource for large language models and aims to help travel companies add AI tools to operations, customer service and efficiency. CEO Omri Morgenshtern says the company is investing in AI research and development. At Pomona College, computer science professor Joseph Osborn emphasizes that AI covers everything from simple algorithms to complex machine-learning systems. He says it has advanced but remains limited and far from human-like intelligence, and argues that understanding its principles can help people solve real-world problems.

Key Takeaways

  • A 12-GPU used Nvidia V100 server costing $9,450 ran DeepSeek V4.1 Flash experts, returning its first code token in 58 seconds on a 256K context.
  • A $1,042 three-V100 system ran Qwen 3.6 35B at 72.7 tokens per second with a 128K context, 19% faster than an M5 Max Mac Studio in the reported comparison.
  • RAM and quantization shape what models run locally: 16 GB can support 4-bit Qwen 3 8B and Llama 3.1 8B; 8 GB supports smaller models including Llama 3.2 3B.
  • The FBI’s Internet Crime Complaint Center reported over $20 billion in losses across more than 1 million complaints in 2025, with over $893 million linked to AI-related cybercrime.
  • A GAO report says AI can speed up cyberattacks, while autonomous agents may perform several attack stages with minimal human input; it also identifies risks including prompt injection and weak security testing.
  • Only 1% of surveyed security leaders are comfortable with AI agents handling most tasks without human approval today; 55% worry AI attacks could outpace their teams.
  • A VB Pulse survey found 46% of enterprises had an AI-agent incident or near miss in the past year. Distinct agent identities were associated with faster reported shutdown capability than shared keys.
  • A Microsoft software-update problem on Nov. 12 grounded planes and disrupted banking systems; the article says the outage was not caused by a cyberattack.
  • Marsh found a substantial AI training gap: 78% of executives reported employer-provided training, compared with 32% of entry-level professionals and 20% of hourly workers.
  • Booking.com’s B2B division plans to help travel companies adopt AI tools and is investing in AI research and development.

Lawyer’s V100 server speeds up DeepSeek responses on long documents

A lawyer’s $9,450 server uses 12 used Nvidia V100 GPUs to run DeepSeek V4.1 Flash experts with 4-bit NVFP4. With updated power and clock settings, it produced the first code token after 58 seconds on a 256K context, compared with 108.4 seconds after the previous upgrade. Prompt reading exceeded 4,500 tokens per second past 32K, and three simultaneous requests reached a combined 360 tokens per second for code and 213 for legal tasks. The builder says the system is now his default DeepSeek machine, but notes that the tests do not measure answer quality and plans to study batched decoding.

Three used Nvidia V100s reach 100 tokens per second on Qwen

A $1,042 homelab system with three used Nvidia V100 GPUs ran the Qwen 3.6 35B model at 100.8 tokens per second with a 4K context and 72.7 at 128K. The V100 setup fell slightly behind an M5 Max Mac Studio at shorter contexts but was 19% faster at 128K, though the systems used different software. A two V100 setup and a 4-bit model reached 77.1 tokens per second at 128K. The builder also compared older P100 GPUs and newer H100 and B200 systems, and plans to test different inference software and speculative decoding.

Former FBI investigator warns companies may miss rogue AI agents

Paul Bingham, a former FBI cyber investigator and senior vice president and executive dean at Western Governors University, warned that many companies cannot recognize when an AI agent acts in an unusual way. He said businesses often give agents credentials before limiting what they can access, and recommended restricting access, learning normal agent behavior, and training staff. A VB Pulse survey found that 46% of enterprises reported an agent incident or near miss in the past year. Among companies that gave agents their own identities, 60% said they could shut down a rogue agent within minutes, compared with 33% of companies using shared keys. Bingham also said agents should not approve multi-factor authentication prompts on their own.

Microsoft outage shows risks of growing dependence on technology

A Microsoft outage on Nov. 12 grounded planes and disrupted banking systems after a software update problem, according to the article. It was not caused by a cyberattack, but the disruption showed how vulnerable daily services can be when they depend heavily on technology. The article warns that attacks can take hours to carry out and months to fix, and says adversaries are already inside U.S. networks. It argues that the AI age brings risks from malicious uses of technology, including autonomous AI hackers, and calls for stronger defenses.

AI agents are speeding up cyberattacks with little human help

A Government Accountability Office report says AI can make phishing and deepfake content more convincing and speed up each stage of cyberattacks, reducing reconnaissance from weeks to hours or less. Autonomous AI agents are an emerging concern because they can carry out several attack stages with minimal human input. The FBI’s Internet Crime Complaint Center reported more than $20 billion in losses across over 1 million complaints in 2025, including more than $893 million linked to AI-related cybercrime. The GAO also highlighted risks such as prompt injection, weak ways to test AI security tools, fragmented regulations, high or unpredictable costs, and a shortage of trained workers. The report says AI can help defenders too, but its speed advantage currently favors attackers.

Security teams weigh how much power to give AI agents

A survey found that 55% of security leaders worry AI-driven attacks could outpace their teams’ ability to respond. Security analysts have limited time, and each alert can require checking several systems before deciding what happened and whether it matters. Only 1% of respondents are comfortable letting AI agents act on most tasks without human approval today, and just 7% expect to be comfortable with that level of autonomy in a year. Most teams want agents to take on more work, but not every action.

AI training gap between executives and workers may slow adoption

A Marsh report released Wednesday found that executives receive more AI training than lower-level workers. In a survey of 6,700 employees across 10 countries, 78% of executives and 44% of experienced professionals said their employers offered AI training in the past year, compared with 32% of entry-level professionals and 20% of hourly workers. Regular AI use also varied, with 64% of executives using AI regularly, compared with 20% of entry-level respondents. Overall, 37% of employees used AI regularly, even though 71% had used it at least once. Marsh Global Workforce Leader Ravin Jesuthasan said the training gap may contribute to a “stalled transformation.”

Pomona professor explains what AI can and cannot do

Joseph Osborn, an associate professor of computer science at Pomona College, discussed how AI works and how he teaches the subject. He said AI is not one single technology, but includes methods ranging from simple algorithms to complex machine learning systems. Osborn uses real-world examples to help students understand AI’s underlying ideas rather than rely on hype. He said AI has advanced but remains limited and is still far from human-like intelligence, while people remain better in many areas. He is optimistic that understanding AI’s principles can help people solve real-world problems.

Booking.com unit aims to power AI tools for travel companies

Omri Morgenshtern, CEO of Booking Partner Division and Agoda, discussed the company’s new B2B unit and growth plans at The Phocuswright Conference. Booking.com’s B2B arm is positioning itself as an AI resource for large language models. It aims to help travel companies add AI-powered solutions to their operations. The company says these tools can improve customer service and efficiency, and help businesses compete. Booking is investing in AI research and development to support the effort.

How much RAM you need to run AI models on your computer

A computer with 16 GB of RAM can run 4-bit quantized models such as Qwen 3 8B and Llama 3.1 8B. With 8 GB, choices are more limited and include Llama 3.2 3B, Qwen 3 4B, and Mistral 7B with 4-bit quantization. Quantization reduces a model’s size and memory use without greatly affecting performance or accuracy. More RAM can support larger models, while memory bandwidth affects how responsive they feel. Leave memory available for the operating system and other apps, since a model also uses RAM while running.

Sources

NOTE:

This news brief was generated using AI technology (including, but not limited to, Google Gemini API, Llama, Grok, and Mistral) from aggregated news articles, with minimal to no human editing/review. It is provided for informational purposes only and may contain inaccuracies or biases. This is not financial, investment, or professional advice. If you have any questions or concerns, please verify all information with the linked original articles in the Sources section below.

AI Local AI inference Used Nvidia V100 GPUs AI model benchmarking DeepSeek Qwen Llama Model quantization GPU servers AI cybersecurity AI-powered phishing Deepfakes Autonomous AI agents Prompt injection Agent identity and access control AI security incidents AI governance AI workforce training Enterprise AI adoption Travel technology

Comments

Loading...