EADST

GGML Q4_0 Quantize Analysis in llama.cpp

GGML Q4_0 Quantization in llama.cpp

For the LLAMA7B model, there are 387 tensors consisting of various weights and biases. These tensors include token_embd.weight, 32 sets of attention and feedforward network weights and biases (attn_norm.weigh, attn_q.weight, attn_k.weight, attn_v.weight, attn_q.bias, attn_k.bias, attn_v.bias, attn_output.weight, ffn_norm.weight, ffn_up.weight, ffn_gate.weight, ffn_down.weight), output_norm.weight, and output.weight.

Quantization Details:

  • Total Tensors for Quantization: 226
  • token_embd.weight
  • 32 sets of: attn_q.weight, attn_k.weight, attn_v.weight, attn_output.weight, ffn_up.weight, ffn_gate.weight, ffn_down.weight
  • output.weight

Tensor Breakdown:

  • llama_model_loader:
  • f32 type: 161 tensors
  • f16 type: 226 tensors
  • llama_model_quantize_internal:
  • Meta size: 6162784 bytes

Example Tensors:

  • [ 1/ 387] token_embd.weight - [ 4096, 151851, 1, 1], type = f16, quantizing to q4_0 .. size = 1186.34 MB -> 333.66 MB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

[ 2/ 387] blk.0.attn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 3/ 387] blk.0.attn_q.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020

  • [ 4/ 387] blk.0.attn_k.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.076 0.055 0.037 0.024 0.020

  • [ 5/ 387] blk.0.attn_v.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021

[ 6/ 387] blk.0.attn_q.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

[ 7/ 387] blk.0.attn_k.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

[ 8/ 387] blk.0.attn_v.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 9/ 387] blk.0.attn_output.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021

[ 10/ 387] blk.0.ffn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 11/ 387] blk.0.ffn_up.weight - [ 4096, 11008, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021

  • [ 12/ 387] blk.0.ffn_gate.weight - [ 4096, 11008, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.040 0.026 0.021

  • [ 13/ 387] blk.0.ffn_down.weight - [11008, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

*...and so on for other tensors [14/ 387]-[385/ 387] *

The remaining 31 blocks follow a similar pattern. blk.0*-blk.31*

[ 386/ 387] output_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 387/ 387] output.weight - [ 4096, 151851, 1, 1], type = f16, quantizing to q6_K .. size = 1186.34 MB -> 486.58 MB | hist:

llama_model_quantize_internal: model size = 14727.19 MB

llama_model_quantize_internal: quant size = 4296.76 MB

llama_model_quantize_internal: hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

Reference:

相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
Sklearn Pillow Animate Jupyter 长期记忆 自我改进 Land Ubuntu API LLM Template 多线程 BTC git Conda Tracking Hungarian logger Heatmap Permission Augmentation Cloudreve Michelin Jetson Quantization Bitcoin DeepSeek GIT Dataset Miniforge Agent Use Zip 继承 Math 签证 SQLite Image2Text Google Plotly TSV PyCharm OCR COCO CSV CLAP WAN FP16 Interview 图标 IndexTTS2 Clash Disk ModelScope 腾讯云 git-lfs Qwen2.5 OpenAI Firewall TensorFlow 多进程 Statistics 算法题 Numpy GPT4 Claude 版权 FP32 Password Food 论文速读 ResNet-50 LLAMA Mixtral 飞书 Card 关于博主 RL Llama uwsgi Translation Streamlit Hotel v2ray Logo LaTeX 证件照 Domain icon Input CC Shortcut FP8 Knowledge NLTK JSON Docker 净利润 Git XML ms-swift GoogLeNet torchinfo Rebuttal Plate Review 论文 Tensor 财报 Bipartite News GGML Algorithm tqdm 图形思考法 VGG-16 Transformers CUDA PDB tar FP64 DeepStream Search Vmess FastAPI uWSGI Django Hilton CTC 强化学习 云服务器 Website v0.dev MD5 Pickle GPTQ Github Paddle Python Web UNIX Datetime CAM Windows Paper WebCrawler Pandas Video Color scipy QWEN BF16 SVR 第一性原理 YOLO Random VSCode Nginx HuggingFace 公式 Proxy RAR SAM llama.cpp NameSilo Baidu Qwen2 Breakpoint Distillation ChatGPT EXCEL Jev UI PyTorch Gemma C++ Bert OpenCV Bin FlashAttention InvalidArgumentError TensorRT TTS Anaconda 递归学习法 printf Base64 音频 VPN LeetCode PDF PIP Quantize NLP Data Magnet 域名 搞笑 Markdown hf transformers 个人博客 Linux Tiktoken Qwen Harness 报税 Diagram RGB diffusers AI API网关 SQL Attention 顶会 Vim Freesound CEIR Pytorch Crawler BeautifulSoup 阿里云 mmap XGBoost CV LoRA Ptyhon SPIE HaggingFace Safetensors Excel ONNX
站点统计

本站现有博文340篇,共被浏览1017620次

本站已经建立2718天!

热门文章
文章归档
回到顶部