@tduncan / Nvidia-Optimization-Stack-VLLM Mirror · read-only / commits / 27a7b070db
Fork 0

A high-throughput and memory-efficient inference and serving engine for LLMs

Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git

Synced Nov 17, 04:15 UTC

Add document for vllm paged attention kernel. (#2978)

Jialun Lyu committed Mar 4, 2024 at 09:23 UTC 27a7b070db526326ede3335fb07c1fa13ac008bb
9 files changed +526
docs/source/assets/kernel/k_vecs.png
Binary files /dev/null and b/docs/source/assets/kernel/k_vecs.png differ
docs/source/assets/kernel/key.png
Binary files /dev/null and b/docs/source/assets/kernel/key.png differ
docs/source/assets/kernel/logits_vec.png
Binary files /dev/null and b/docs/source/assets/kernel/logits_vec.png differ
docs/source/assets/kernel/q_vecs.png
Binary files /dev/null and b/docs/source/assets/kernel/q_vecs.png differ
docs/source/assets/kernel/query.png
Binary files /dev/null and b/docs/source/assets/kernel/query.png differ
docs/source/assets/kernel/v_vec.png
Binary files /dev/null and b/docs/source/assets/kernel/v_vec.png differ
docs/source/assets/kernel/value.png
Binary files /dev/null and b/docs/source/assets/kernel/value.png differ
docs/source/dev/kernel/paged_attention.rst new
+525
@@ -0,0 +1,525 @@
1 +vLLM Paged Attention
2 +====================
3 +
4 +- Currently, vLLM utilizes its own implementation of a multi-head query
5 + attention kernel (``csrc/attention/attention_kernels.cu``).
6 + This kernel is designed to be compatible with
7 + vLLM's paged KV caches, where the key and value cache are stored in
8 + separate blocks (note that this block concept differs from the GPU
9 + thread block. So in a later document, I will refer to vLLM paged
10 + attention block as "block", while refer to GPU thread block as
11 + "thread block").
12 +- To achieve high performance, this kernel relies on a specially
13 + designed memory layout and access method, specifically when threads
14 + read data from global memory to shared memory. The purpose of this
15 + document is to provide a high-level explanation of the kernel
16 + implementation step by step, aiding those who wish to learn about the
17 + vLLM multi-head query attention kernel. After going through this
18 + document, users will likely have a better understanding and feel easier
19 + to follow the actual implementation.
20 +- Please note that this document may not cover all details, such as how
21 + to calculate the correct index for the corresponding data or the dot
22 + multiplication implementation. However, after reading this document
23 + and becoming familiar with the high-level logic flow, it should be
24 + easier for you to read the actual code and understand the details.
25 +
26 +Inputs
27 +------
28 +
29 +- The kernel function takes a list of arguments for the current thread
30 + to perform its assigned work. The three most important arguments are
31 + the input pointers ``q``, ``k_cache``, and ``v_cache``, which point
32 + to query, key, and value data on global memory that need to be read
33 + and processed. The output pointer ``out`` points to global memory
34 + where the result should be written. These four pointers actually
35 + refer to multi-dimensional arrays, but each thread only accesses the
36 + portion of data assigned to it. I have omitted all other runtime
37 + parameters here for simplicity.
38 +
39 + .. code:: cpp
40 +
41 + template<
42 + typename scalar_t,
43 + int HEAD_SIZE,
44 + int BLOCK_SIZE,
45 + int NUM_THREADS,
46 + int PARTITION_SIZE = 0>
47 + __device__ void paged_attention_kernel(
48 + ... // Other side args.
49 + const scalar_t* __restrict__ out, // [num_seqs, num_heads, max_num_partitions, head_size]
50 + const scalar_t* __restrict__ q, // [num_seqs, num_heads, head_size]
51 + const scalar_t* __restrict__ k_cache, // [num_blocks, num_kv_heads, head_size/x, block_size, x]
52 + const scalar_t* __restrict__ v_cache, // [num_blocks, num_kv_heads, head_size, block_size]
53 + ... // Other side args.
54 + )
55 +
56 +- There are also a list of template arguments above the function
57 + signature that are determined during compilation time. ``scalar_t``
58 + represents the data type of the query, key, and value data elements,
59 + such as FP16. ``HEAD_SIZE`` indicates the number of elements in each
60 + head. ``BLOCK_SIZE`` refers to the number of tokens in each block.
61 + ``NUM_THREADS`` denotes the number of threads in each thread block.
62 + ``PARTITION_SIZE`` represents the number of tensor parallel GPUs (For
63 + simplicity, we assume this is 0 and tensor parallel is disabled).
64 +- With these arguments, we need to perform a sequence of preparations.
65 + This includes calculating the current head index, block index, and
66 + other necessary variables. However, for now, we can ignore these
67 + preparations and proceed directly to the actual calculations. It will
68 + be easier to understand them once we grasp the entire flow.
69 +
70 +Concepts
71 +--------
72 +
73 +- Just before we dive into the calculation flow, I want to describe a
74 + few concepts that are needed for later sections. However, you may
75 + skip this section and return later if you encounter any confusing
76 + terminologies.
77 +- **Sequence**: A sequence represents a client request. For example,
78 + the data pointed to by ``q`` has a shape of
79 + ``[num_seqs, num_heads, head_size]``. That represents there are total
80 + ``num_seqs`` of query sequence data are pointed by ``q``. Since this
81 + kernel is a single query attention kernel, each sequence only has one
82 + query token. Hence, the ``num_seqs`` equals the total number of tokens
83 + that are processed in the batch.
84 +- **Context**: The context consists of the generated tokens from the
85 + sequence. For instance, ``["What", "is", "your"]`` are the context
86 + tokens, and the input query token is ``"name"``. The model might
87 + generate the token ``"?"``.
88 +- **Vec**: The vec is a list of elements that are fetched and
89 + calculated together. For query and key data, the vec size
90 + (``VEC_SIZE``) is determined so that each thread group can fetch and
91 + calculate 16 bytes of data at a time. For value data, the vec size
92 + (``V_VEC_SIZE``) is determined so that each thread can fetch and
93 + calculate 16 bytes of data at a time. For example, if the
94 + ``scalar_t`` is FP16 (2 bytes) and ``THREAD_GROUP_SIZE`` is 2, the
95 + ``VEC_SIZE`` will be 4, while the ``V_VEC_SIZE`` will be 8.
96 +- **Thread group**: The thread group is a small group of
97 + threads(\ ``THREAD_GROUP_SIZE``) that fetches and calculates one
98 + query token and one key token at a time. Each thread handles only a
99 + portion of the token data. The total number of elements processed by
100 + one thread group is referred as ``x``. For example, if the thread
101 + group contains 2 threads and the head size is 8, then thread 0
102 + handles the query and key elements at index 0, 2, 4, 6, while thread
103 + 1 handles the elements at index 1, 3, 5, 7.
104 +- **Block**: The key and value cache data in vLLM are split into
105 + blocks. Each block stores data for a fixed number(\ ``BLOCK_SIZE``)
106 + of tokens at one head. Each block may contain only a portion of the
107 + whole context tokens. For example, if the block size is 16 and the
108 + head size is 128, then for one head, one block can store 16 \* 128 =
109 + 2048 elements.
110 +- **Warp**: A warp is a group of 32 threads(\ ``WARP_SIZE``) that
111 + execute simultaneously on a stream multiprocessor (SM). In this
112 + kernel, each warp processes the calculation between one query token
113 + and key tokens of one entire block at a time (it may process multiple
114 + blocks in multiple iterations). For example, if there are 4 warps and
115 + 6 blocks for one context, the assignment would be like warp 0 handles
116 + the 0th, 4th blocks, warp 1 handles the 1st, 5th blocks, warp 2
117 + handles the 2nd block and warp 3 handles the 3rd block.
118 +- **Thread block**: A thread block is a group of
119 + threads(\ ``NUM_THREADS``) that can access the same shared memory.
120 + Each thread block contains multiple warps(\ ``NUM_WARPS``), and in
121 + this kernel, each thread block processes the calculation between one
122 + query token and key tokens of a whole context.
123 +- **Grid**: A grid is a collection of thread blocks and defines the
124 + shape of the collection. In this kernel, the shape is
125 + ``(num_heads, num_seqs, max_num_partitions)``. Therefore, each thread
126 + block only handles the calculation for one head, one sequence, and
127 + one partition.
128 +
129 +Query
130 +-----
131 +
132 +- This section will introduce how query data is stored in memory and
133 + fetched by each thread. As mentioned above, each thread group fetches
134 + one query token data, while each thread itself only handles a part of
135 + one query token data. Within each warp, every thread group will fetch
136 + the same query token data, but will multiply it with different key
137 + token data.
138 +
139 + .. code:: cpp
140 +
141 + const scalar_t* q_ptr = q + seq_idx * q_stride + head_idx * HEAD_SIZE;
142 +
143 + .. figure:: ../../assets/kernel/query.png
144 + :alt: query
145 + :width: 70%
146 + :align: center
147 +
148 + Query data of one token at one head
149 +
150 +- Each thread defines its own ``q_ptr`` which points to the assigned
151 + query token data on global memory. For example, if ``VEC_SIZE`` is 4
152 + and ``HEAD_SIZE`` is 128, the ``q_ptr`` points to data that contains
153 + total of 128 elements divided into 128 / 4 = 32 vecs.
154 +
155 + .. figure:: ../../assets/kernel/q_vecs.png
156 + :alt: q_vecs
157 + :width: 70%
158 + :align: center
159 +
160 + ``q_vecs`` for one thread group
161 +
162 + .. code:: cpp
163 +
164 + __shared__ Q_vec q_vecs[THREAD_GROUP_SIZE][NUM_VECS_PER_THREAD];
165 +
166 +- Next, we need to read the global memory data pointed to by ``q_ptr``
167 + into shared memory as ``q_vecs``. It is important to note that each
168 + vecs is assigned to a different row. For example, if the
169 + ``THREAD_GROUP_SIZE`` is 2, thread 0 will handle the 0th row vecs,
170 + while thread 1 handles the 1st row vecs. By reading the query data in
171 + this way, neighboring threads like thread 0 and thread 1 can read
172 + neighbor memory, achieving the memory coalescing to improve
173 + performance.
174 +
175 +Key
176 +---
177 +
178 +- Similar to the "Query" section, this section introduces memory layout
179 + and assignment for keys. While each thread group only handle one
180 + query token one kernel run, it may handle multiple key tokens across
181 + multiple iterations. Meanwhile, each warp will process multiple blocks
182 + of key tokens in multiple iterations, ensuring that all context
183 + tokens are processed by the entire thread group after the kernel run.
184 + In this context, "handle" refers to performing the dot multiplication
185 + between query data and key data.
186 +
187 + .. code:: cpp
188 +
189 + const scalar_t* k_ptr = k_cache + physical_block_number * kv_block_stride
190 + + kv_head_idx * kv_head_stride
191 + + physical_block_offset * x;
192 +
193 +- Unlike to ``q_ptr``, ``k_ptr`` in each thread will point to different
194 + key token at different iterations. As shown above, that ``k_ptr``
195 + points to key token data based on ``k_cache`` at assigned block,
196 + assigned head and assigned token.
197 +
198 + .. figure:: ../../assets/kernel/key.png
199 + :alt: key
200 + :width: 70%
201 + :align: center
202 +
203 + Key data of all context tokens at one head
204 +
205 +- The diagram above illustrates the memory layout for key data. It
206 + assumes that the ``BLOCK_SIZE`` is 16, ``HEAD_SIZE`` is 128, ``x`` is
207 + 8, ``THREAD_GROUP_SIZE`` is 2, and there are a total of 4 warps. Each
208 + rectangle represents all the elements for one key token at one head,
209 + which will be processed by one thread group. The left half shows the
210 + total 16 blocks of key token data for warp 0, while the right half
211 + represents the remaining key token data for other warps or
212 + iterations. Inside each rectangle, there are a total 32 vecs (128
213 + elements for one token) that will be processed by 2 threads (one
214 + thread group) separately.
215 +
216 + .. figure:: ../../assets/kernel/k_vecs.png
217 + :alt: k_vecs
218 + :width: 70%
219 + :align: center
220 +
221 + ``k_vecs`` for one thread
222 +
223 + .. code:: cpp
224 +
225 + K_vec k_vecs[NUM_VECS_PER_THREAD]
226 +
227 +- Next, we need to read the key token data from ``k_ptr`` and store
228 + them on register memory as ``k_vecs``. We use register memory for
229 + ``k_vecs`` because it will only be accessed by one thread once,
230 + whereas ``q_vecs`` will be accessed by multiple threads multiple
231 + times. Each ``k_vecs`` will contain multiple vectors for later
232 + calculation. Each vec will be set at each inner iteration. The
233 + assignment of vecs allows neighboring threads in a warp to read
234 + neighboring memory together, which again promotes the memory
235 + coalescing. For instance, thread 0 will read vec 0, while thread 1
236 + will read vec 1. In the next inner loop, thread 0 will read vec 2,
237 + while thread 1 will read vec 3, and so on.
238 +- You may still be a little confused about the overall flow. Don't
239 + worry, please keep reading the next "QK" section. It will illustrate
240 + the query and key calculation flow in a clearer and higher-level
241 + manner.
242 +
243 +QK
244 +---
245 +
246 +- As shown the pseudo code below, before the entire for loop block, we
247 + fetch the query data for one token and store it in ``q_vecs``. Then,
248 + in the outer for loop, we iterate through different ``k_ptrs`` that
249 + point to different tokens and prepare the ``k_vecs`` in the inner for
250 + loop. Finally, we perform the dot multiplication between the
251 + ``q_vecs`` and each ``k_vecs``.
252 +
253 + .. code:: cpp
254 +
255 + q_vecs = ...
256 + for ... {
257 + k_ptr = ...
258 + for ... {
259 + k_vecs[i] = ...
260 + }
261 + ...
262 + float qk = scale * Qk_dot<scalar_t, THREAD_GROUP_SIZE>::dot(q_vecs[thread_group_offset], k_vecs);
263 + }
264 +
265 +- As mentioned before, for each thread, it only fetches part of the
266 + query and key token data at a time. However, there will be a cross
267 + thread group reduction happen in the ``Qk_dot<>::dot`` . So ``qk``
268 + returned here is not just between part of the query and key token dot
269 + multiplication, but actually a full result between entire query and
270 + key token data.
271 +- For example, if the value of ``HEAD_SIZE`` is 128 and
272 + ``THREAD_GROUP_SIZE`` is 2, each thread's ``k_vecs`` will contain
273 + total 64 elements. However, the returned ``qk`` is actually the
274 + result of dot multiplication between 128 query elements and 128 key
275 + elements. If you want to learn more about the details of the dot
276 + multiplication and reduction, you may refer to the implementation of
277 + ``Qk_dot<>::dot``. However, for the sake of simplicity, I will not
278 + cover it in this document.
279 +
280 +Softmax
281 +-------
282 +
283 +- Next, we need to calculate the normalized softmax for all ``qk``\ s,
284 + as shown above, where each :math:`x` represents a ``qk``. To do this,
285 + we must obtain the reduced value of ``qk_max``\ (:math:`m(x)`) and
286 + the ``exp_sum``\ (:math:`\ell(x)`) of all ``qk``\ s. The reduction
287 + should be performed across the entire thread block, encompassing
288 + results between the query token and all context key tokens.
289 +
290 + .. math::
291 + :nowrap:
292 +
293 + \begin{gather*}
294 + m(x):=\max _i \quad x_i \\ \quad f(x):=\left[\begin{array}{lll}e^{x_1-m(x)} & \ldots & e^{x_B-m(x)}\end{array}\right]\\ \quad \ell(x):=\sum_i f(x)_i \\
295 + \quad \operatorname{softmax}(x):=\frac{f(x)}{\ell(x)}
296 + \end{gather*}
297 +
298 +``qk_max`` and ``logits``
299 +~~~~~~~~~~~~~~~~~~~~~~~~~
300 +
301 +- Just right after we get the ``qk`` result, we can set the temporary
302 + ``logits`` result with ``qk`` (In the end, the ``logits`` should
303 + store the normalized softmax result). Also we can compare and collect
304 + the ``qk_max`` for all ``qk``\ s that are calculated by current
305 + thread group.
306 +
307 + .. code:: cpp
308 +
309 + if (thread_group_offset == 0) {
310 + const bool mask = token_idx >= context_len;
311 + logits[token_idx - start_token_idx] = mask ? 0.f : qk;
312 + qk_max = mask ? qk_max : fmaxf(qk_max, qk);
313 + }
314 +
315 +- Please note that the ``logits`` here is on shared memory, so each
316 + thread group will set the fields for its own assigned context tokens.
317 + Overall, the size of logits should be number of context tokens.
318 +
319 + .. code:: cpp
320 +
321 + for (int mask = WARP_SIZE / 2; mask >= THREAD_GROUP_SIZE; mask /= 2) {
322 + qk_max = fmaxf(qk_max, VLLM_SHFL_XOR_SYNC(qk_max, mask));
323 + }
324 +
325 + if (lane == 0) {
326 + red_smem[warp_idx] = qk_max;
327 + }
328 +
329 +- Then we need to get the reduced ``qk_max`` across each warp. The main
330 + idea is to make threads in warp to communicate with each other and
331 + get the final max ``qk`` .
332 +
333 + .. code:: cpp
334 +
335 + for (int mask = NUM_WARPS / 2; mask >= 1; mask /= 2) {
336 + qk_max = fmaxf(qk_max, VLLM_SHFL_XOR_SYNC(qk_max, mask));
337 + }
338 + qk_max = VLLM_SHFL_SYNC(qk_max, 0);
339 +
340 +- Finally, we can get the reduced ``qk_max`` from whole thread block by
341 + compare the ``qk_max`` from all warps in this thread block. Then we
342 + need to broadcast the final result to each thread.
343 +
344 +``exp_sum``
345 +~~~~~~~~~~~
346 +
347 +- Similar to ``qk_max``, we need to get the reduced sum value from the
348 + entire thread block too.
349 +
350 + .. code:: cpp
351 +
352 + for (int i = thread_idx; i < num_tokens; i += NUM_THREADS) {
353 + float val = __expf(logits[i] - qk_max);
354 + logits[i] = val;
355 + exp_sum += val;
356 + }
357 + ...
358 + exp_sum = block_sum<NUM_WARPS>(&red_smem[NUM_WARPS], exp_sum);
359 +
360 +- Firstly, sum all exp values from each thread group, and meanwhile,
361 + convert each entry of ``logits`` from ``qk`` to ``exp(qk - qk_max)``.
362 + Please note, the ``qk_max`` here is already the max ``qk`` across the
363 + whole thread block. And then we can do reduction for ``exp_sum``
364 + across whole thread block just like the ``qk_max``.
365 +
366 + .. code:: cpp
367 +
368 + const float inv_sum = __fdividef(1.f, exp_sum + 1e-6f);
369 + for (int i = thread_idx; i < num_tokens; i += NUM_THREADS) {
370 + logits[i] *= inv_sum;
371 + }
372 +
373 +- Finally, with the reduced ``qk_max`` and ``exp_sum``, we can obtain
374 + the final normalized softmax result as ``logits``. This ``logits``
375 + variable will be used for dot multiplication with the value data in
376 + later steps. Now, it should store the normalized softmax result of
377 + ``qk`` for all assigned context tokens.
378 +
379 +Value
380 +-----
381 +
382 +.. figure:: ../../assets/kernel/value.png
383 + :alt: value
384 + :width: 70%
385 + :align: center
386 +
387 + Value data of all context tokens at one head
388 +
389 +.. figure:: ../../assets/kernel/logits_vec.png
390 + :alt: logits_vec
391 + :width: 50%
392 + :align: center
393 +
394 + ``logits_vec`` for one thread
395 +
396 +.. figure:: ../../assets/kernel/v_vec.png
397 + :alt: v_vec
398 + :width: 70%
399 + :align: center
400 +
401 + List of ``v_vec`` for one thread
402 +
403 +- Now we need to retrieve the value data and perform dot multiplication
404 + with ``logits``. Unlike query and key, there is no thread group
405 + concept for value data. As shown in diagram, different from key token
406 + memory layout, elements from the same column correspond to the same
407 + value token. For one block of value data, there are ``HEAD_SIZE`` of
408 + rows and ``BLOCK_SIZE`` of columns that are split into multiple
409 + ``v_vecs``.
410 +- Each thread always fetches ``V_VEC_SIZE`` elements from the same
411 + ``V_VEC_SIZE`` of tokens at a time. As a result, a single thread
412 + retrieves multiple ``v_vec``\ s from different rows and the same
413 + columns through multiple inner iterations. For each ``v_vec``, it
414 + needs to be dot multiplied with the corresponding ``logits_vec``,
415 + which is also ``V_VEC_SIZE`` elements from ``logits``. Overall, with
416 + multiple inner iterations, each warp will process one block of value
417 + tokens. And with multiple outer iterations, the whole context value
418 + tokens are processd
419 +
420 + .. code:: cpp
421 +
422 + float accs[NUM_ROWS_PER_THREAD];
423 + for ... { // Iteration over different blocks.
424 + logits_vec = ...
425 + for ... { // Iteration over different rows.
426 + v_vec = ...
427 + ...
428 + accs[i] += dot(logits_vec, v_vec);
429 + }
430 + }
431 +
432 +- As shown in the above pseudo code, in the outer loop, similar to
433 + ``k_ptr``, ``logits_vec`` iterates over different blocks and reads
434 + ``V_VEC_SIZE`` elements from ``logits``. In the inner loop, each
435 + thread reads ``V_VEC_SIZE`` elements from the same tokens as a
436 + ``v_vec`` and performs dot multiplication. It is important to note
437 + that in each inner iteration, the thread fetches different head
438 + position elements for the same tokens. The dot result is then
439 + accumulated in ``accs``. Therefore, each entry of ``accs`` is mapped
440 + to a head position assigned to the current thread.
441 +- For example, if ``BLOCK_SIZE`` is 16 and ``V_VEC_SIZE`` is 8, each
442 + thread fetches 8 value elements for 8 tokens at a time. Each element
443 + is from different tokens at the same head position. If ``HEAD_SIZE``
444 + is 128 and ``WARP_SIZE`` is 32, for each inner loop, a warp needs to
445 + fetch ``WARP_SIZE * V_VEC_SIZE = 256`` elements. This means there are
446 + a total of 128 \* 16 / 256 = 8 inner iterations for a warp to handle
447 + a whole block of value tokens. And each ``accs`` in each thread
448 + contains 8 elements that accumulated at 8 different head positions.
449 + For the thread 0, the ``accs`` variable will have 8 elements, which
450 + are 0th, 16th … 112th elements of a value head that are accumulated
451 + from all assigned 8 tokens.
452 +
453 +LV
454 +---
455 +- Now, we need to perform reduction for ``accs`` within each warp. This
456 + process allows each thread to accumulate the ``accs`` for the
457 + assigned head positions of all tokens in one block.
458 +
459 + .. code:: cpp
460 +
461 + for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
462 + float acc = accs[i];
463 + for (int mask = NUM_V_VECS_PER_ROW / 2; mask >= 1; mask /= 2) {
464 + acc += VLLM_SHFL_XOR_SYNC(acc, mask);
465 + }
466 + accs[i] = acc;
467 + }
468 +
469 +- Next, we perform reduction for ``accs`` across all warps, allowing
470 + each thread to have the accumulation of ``accs`` for the assigned
471 + head positions of all context tokens. Please note that each ``accs``
472 + in every thread only stores the accumulation for a portion of
473 + elements of the entire head for all context tokens. However, overall,
474 + all results for output have been calculated but are just stored in
475 + different thread register memory.
476 +
477 + .. code:: cpp
478 +
479 + float* out_smem = reinterpret_cast<float*>(shared_mem);
480 + for (int i = NUM_WARPS; i > 1; i /= 2) {
481 + // Upper warps write to shared memory.
482 + ...
483 + float* dst = &out_smem[(warp_idx - mid) * HEAD_SIZE];
484 + for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
485 + ...
486 + dst[row_idx] = accs[i];
487 + }
488 +
489 + // Lower warps update the output.
490 + const float* src = &out_smem[warp_idx * HEAD_SIZE];
491 + for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
492 + ...
493 + accs[i] += src[row_idx];
494 + }
495 +
496 + // Write out the accs.
497 + }
498 +
499 +Output
500 +------
501 +
502 +- Now we can write all of calculated result from local register memory
503 + to final output global memory.
504 +
505 + .. code:: cpp
506 +
507 + scalar_t* out_ptr = out + seq_idx * num_heads * max_num_partitions * HEAD_SIZE
508 + + head_idx * max_num_partitions * HEAD_SIZE
509 + + partition_idx * HEAD_SIZE;
510 +
511 +- First, we need to define the ``out_ptr`` variable, which points to
512 + the start address of the assigned sequence and assigned head.
513 +
514 + .. code:: cpp
515 +
516 + for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
517 + const int row_idx = lane / NUM_V_VECS_PER_ROW + i * NUM_ROWS_PER_ITER;
518 + if (row_idx < HEAD_SIZE && lane % NUM_V_VECS_PER_ROW == 0) {
519 + from_float(*(out_ptr + row_idx), accs[i]);
520 + }
521 + }
522 +
523 +- Finally, we need to iterate over different assigned head positions
524 + and write out the corresponding accumulated result based on the
525 + ``out_ptr``.
docs/source/index.rst
+1
@@ -98,6 +98,7 @@ Documentation
98 :caption: Developer Documentation
99
100 dev/engine/engine_index
101 + dev/kernel/paged_attention
102
103 Indices and tables
104 ==================