EADST

GGML Q4_0 Quantize Analysis in llama.cpp

GGML Q4_0 Quantization in llama.cpp

For the LLAMA7B model, there are 387 tensors consisting of various weights and biases. These tensors include token_embd.weight, 32 sets of attention and feedforward network weights and biases (attn_norm.weigh, attn_q.weight, attn_k.weight, attn_v.weight, attn_q.bias, attn_k.bias, attn_v.bias, attn_output.weight, ffn_norm.weight, ffn_up.weight, ffn_gate.weight, ffn_down.weight), output_norm.weight, and output.weight.

Quantization Details:

  • Total Tensors for Quantization: 226
  • token_embd.weight
  • 32 sets of: attn_q.weight, attn_k.weight, attn_v.weight, attn_output.weight, ffn_up.weight, ffn_gate.weight, ffn_down.weight
  • output.weight

Tensor Breakdown:

  • llama_model_loader:
  • f32 type: 161 tensors
  • f16 type: 226 tensors
  • llama_model_quantize_internal:
  • Meta size: 6162784 bytes

Example Tensors:

  • [ 1/ 387] token_embd.weight - [ 4096, 151851, 1, 1], type = f16, quantizing to q4_0 .. size = 1186.34 MB -> 333.66 MB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

[ 2/ 387] blk.0.attn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 3/ 387] blk.0.attn_q.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020

  • [ 4/ 387] blk.0.attn_k.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.076 0.055 0.037 0.024 0.020

  • [ 5/ 387] blk.0.attn_v.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021

[ 6/ 387] blk.0.attn_q.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

[ 7/ 387] blk.0.attn_k.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

[ 8/ 387] blk.0.attn_v.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 9/ 387] blk.0.attn_output.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021

[ 10/ 387] blk.0.ffn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 11/ 387] blk.0.ffn_up.weight - [ 4096, 11008, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021

  • [ 12/ 387] blk.0.ffn_gate.weight - [ 4096, 11008, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.040 0.026 0.021

  • [ 13/ 387] blk.0.ffn_down.weight - [11008, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

*...and so on for other tensors [14/ 387]-[385/ 387] *

The remaining 31 blocks follow a similar pattern. blk.0*-blk.31*

[ 386/ 387] output_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 387/ 387] output.weight - [ 4096, 151851, 1, 1], type = f16, quantizing to q6_K .. size = 1186.34 MB -> 486.58 MB | hist:

llama_model_quantize_internal: model size = 14727.19 MB

llama_model_quantize_internal: quant size = 4296.76 MB

llama_model_quantize_internal: hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

Reference:

相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
CLAP RAR Ubuntu NLTK Excel Food TensorRT Datetime RGB Pandas tqdm 净利润 SAM Windows GPTQ GIT DeepSeek CC 算法题 MD5 财报 音频 Base64 Card transformers Website 阿里云 COCO 飞书 Conda Claude Distillation Search TTS FlashAttention 证件照 腾讯云 ModelScope Shortcut CUDA 版权 云服务器 LeetCode Freesound Django FP16 uwsgi PIP Tensor printf 公式 FP64 UI v0.dev PDF NLP Qwen2.5 ChatGPT CV 递归学习法 Translation Data 多进程 FP8 Vim TSV LLAMA NameSilo Password Augmentation InvalidArgumentError Attention OpenAI Paddle LoRA GPT4 Magnet Quantize Math Template 论文 HaggingFace SQL GGML OCR Permission Cloudreve Bert mmap 域名 搞笑 个人博客 Color Review git Gemma ONNX CTC 第一性原理 Dataset hf 强化学习 diffusers 顶会 Use Mixtral 长期记忆 Statistics 多线程 Agent Land WebCrawler SPIE Input WAN Nginx PyCharm FastAPI llama.cpp JSON Docker Git Streamlit Diagram Pytorch Image2Text Video AI IndexTTS2 图形思考法 Hungarian VGG-16 Numpy Github C++ SQLite API网关 SVR git-lfs Crawler CEIR v2ray Harness Paper Qwen Interview 报税 关于博主 YOLO Tiktoken PyTorch Algorithm Firewall Linux Jev LLM 继承 VSCode EXCEL FP32 Bipartite 论文速读 uWSGI PDB ms-swift Transformers Ptyhon Pillow 自我改进 BF16 scipy Llama Pickle Python Proxy Web Bin XML icon Breakpoint BTC Jupyter VPN ResNet-50 QWEN Knowledge UNIX Miniforge Qwen2 logger Hilton CSV OpenCV Animate Vmess DeepStream Jetson Hotel XGBoost GoogLeNet Safetensors Quantization LaTeX Plate 签证 TensorFlow API Google Heatmap Disk tar 图标 HuggingFace Sklearn Michelin Tracking Zip Clash Logo Baidu torchinfo Bitcoin Markdown Domain Rebuttal Plotly BeautifulSoup CAM Random Anaconda News RL
站点统计

本站现有博文340篇,共被浏览1010718次

本站已经建立2716天!

热门文章
文章归档
回到顶部