EADST

Review: H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

Title: [NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Rating: Average, Not Recommended

Paper:H2O Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

Code:https://github.com/FMInference/H2O

Review:

The paper introduces an approach to reduce KV cache in large language models by selecting the top k values as the main KV Cache for the following computing. However, the presentation of Algorithm 1 is complex and lacks clarity, for example, the element 'Gi' not being mentioned in the paper. This obscurity makes it difficult to understand the main idea. Further, the actual method, as revealed by the code, is disappointing, at least for LLAMA. It merely involves applying softmax to the QK matrix for updated scores and then selecting the top k indices for the mask update. This approach seems overly simplistic and does not match the expected sophistication of the proposed solution.

该论文提出了一种减少大型语言模型中KV缓存的方法,通过选择计算中的前k个数值作为主要KV缓存进行后续的计算。然而,算法1的呈现复杂且缺乏清晰度,比如元素'Gi'未提及,使得算法难以理解。进一步查看其代码,实际方法令人失望,至少对于LLAMA模型应用来讲。它只是对QK矩阵应用softmax得到更新的分数,然后选择前k个索引更新掩码。这种方法过于简单,并未达到预期的复杂性和提出的解决方案的深度。

modify_llama.py def local_heavy_hitter_mask

def local_heavy_hitter_mask(attn_weights, heavy_budget):
    # 获取注意力权重的数据类型和序列长度
    dtype_attn_weights = attn_weights.dtype
    seq_length = attn_weights.shape[-1]
    padding_length = 0

    # 对注意力权重应用softmax函数以获得正规化的注意力分布
    tmp_attn = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(dtype_attn_weights)

    # 计算前heavy_budget个令牌的累积注意力分数
    accumulated_attention_score = torch.sum(tmp_attn[:,:,padding_length:heavy_budget+padding_length,:], dim=-2) # (head, keys)
    # 对超出预算的部分分数置零
    accumulated_attention_score[:,:,heavy_budget+padding_length:] = 0
    accumulated_attention_score[:,:,:padding_length] = 0

    # 初始化一个与注意力权重形状相同的零掩码张量
    mask_bottom = torch.zeros_like(attn_weights, dtype=torch.bool)
    # 在掩码中标记重击者区域为True
    mask_bottom[:,:, padding_length:heavy_budget+padding_length, padding_length:heavy_budget+padding_length] = True

    # 遍历序列中的每个令牌
    for token_index in range(heavy_budget+padding_length, seq_length):
        # 计算当前令牌的softmax注意力权重
        tmp_attn_index = nn.functional.softmax(attn_weights[:,:,token_index,:], dim=-1, dtype=torch.float32).to(dtype_attn_weights)
        # 选择前heavy_budget-1个最高的累积注意力分数
        _, tmp_topk_index = accumulated_attention_score.topk(k=heavy_budget-1, dim=-1)
        # 创建当前令牌的零掩码张量
        zeros_index = torch.zeros_like(tmp_attn_index, dtype=torch.bool)
        # 更新当前令牌的掩码,标记最高分数的位置
        mask_bottom_index = zeros_index.scatter(-1, tmp_topk_index, True) # (head, keys)
        mask_bottom_index[:,:, token_index] = True

        # 更新总掩码,将当前令牌的掩码信息添加进去
        mask_bottom[:,:,token_index,:] = mask_bottom_index
        # 更新累积注意力分数
        accumulated_attention_score += tmp_attn_index
        accumulated_attention_score = accumulated_attention_score * mask_bottom_index

    # 返回计算得到的掩码
    return mask_bottom
相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
EXCEL FlashAttention CAM Magnet Hilton Hungarian Qwen2.5 SQL torchinfo 第一性原理 公式 Jev Pandas Crawler Gemma Paddle Llama Qwen hf Mixtral Zip v2ray CEIR Clash GPTQ CLAP OpenCV Password TSV transformers UI News HuggingFace WAN Michelin VSCode Augmentation Firewall Plate NLP Jetson 多进程 Domain PDF icon 长期记忆 Harness BeautifulSoup 证件照 Freesound Website FP8 Google Windows Python Django Color 算法题 Hotel Transformers Tensor Translation Attention Food QWEN Breakpoint TensorFlow AI Datetime ModelScope GIT Docker BTC Land 云服务器 RL SAM HaggingFace FP64 Ubuntu Markdown 论文 SQLite Statistics SPIE Interview Distillation 财报 OpenAI ChatGPT NameSilo 腾讯云 CC Tracking Logo Vmess printf Numpy Card Linux WebCrawler Claude Git 图标 Search Use LoRA DeepSeek Knowledge git FP16 GoogLeNet diffusers Image2Text tqdm ResNet-50 mmap 搞笑 Permission CSV Safetensors SVR XGBoost Shortcut GGML 继承 Web 飞书 Animate Bert 多线程 Conda llama.cpp Heatmap C++ Proxy Pickle CUDA Qwen2 Paper API 关于博主 Pytorch Streamlit LLM Sklearn FP32 LaTeX v0.dev Template Diagram 个人博客 RAR uWSGI 净利润 Dataset Excel API网关 Math Nginx OCR Base64 logger CV uwsgi TTS git-lfs 论文速读 LLAMA Anaconda LeetCode 音频 Baidu Plotly 阿里云 TensorRT RGB Rebuttal YOLO Pillow scipy Disk CTC Algorithm 自我改进 InvalidArgumentError DeepStream Random ms-swift Agent 图形思考法 强化学习 报税 PyTorch Review Ptyhon Cloudreve Quantize Video JSON Bipartite Miniforge PyCharm UNIX 域名 ONNX Bitcoin Jupyter FastAPI BF16 Github XML 递归学习法 Tiktoken Vim MD5 版权 Quantization PDB tar COCO 签证 顶会 Input PIP Bin VGG-16 IndexTTS2 Data NLTK GPT4 VPN
站点统计

本站现有博文340篇,共被浏览1007097次

本站已经建立2715天!

热门文章
文章归档
回到顶部