A high-throughput and memory-efficient inference and serving engine for LLMs
Mirrored from https://github.com/tduncan-lum/Nvidia-Optimization-Stack---VLLM.git
Synced Nov 17, 04:15 UTC
Add document for vllm paged attention kernel. (#2978)
Jialun Lyu committed
Mar 4, 2024 at 09:23 UTC
27a7b070db526326ede3335fb07c1fa13ac008bb
9 files changed
+526
docs/source/assets/kernel/k_vecs.png
Binary files /dev/null and b/docs/source/assets/kernel/k_vecs.png differ
docs/source/assets/kernel/key.png
Binary files /dev/null and b/docs/source/assets/kernel/key.png differ
docs/source/assets/kernel/logits_vec.png
Binary files /dev/null and b/docs/source/assets/kernel/logits_vec.png differ
docs/source/assets/kernel/q_vecs.png
Binary files /dev/null and b/docs/source/assets/kernel/q_vecs.png differ
docs/source/assets/kernel/query.png
Binary files /dev/null and b/docs/source/assets/kernel/query.png differ
docs/source/assets/kernel/v_vec.png
Binary files /dev/null and b/docs/source/assets/kernel/v_vec.png differ
docs/source/assets/kernel/value.png
Binary files /dev/null and b/docs/source/assets/kernel/value.png differ
docs/source/dev/kernel/paged_attention.rst
new
+525
@@ -0,0 +1,525 @@
1
+vLLM Paged Attention
2
+====================
3
+
4
+- Currently, vLLM utilizes its own implementation of a multi-head query
5
+ attention kernel (``csrc/attention/attention_kernels.cu``).
6
+ This kernel is designed to be compatible with
7
+ vLLM's paged KV caches, where the key and value cache are stored in
8
+ separate blocks (note that this block concept differs from the GPU
9
+ thread block. So in a later document, I will refer to vLLM paged
10
+ attention block as "block", while refer to GPU thread block as
11
+ "thread block").
12
+- To achieve high performance, this kernel relies on a specially
13
+ designed memory layout and access method, specifically when threads
14
+ read data from global memory to shared memory. The purpose of this
15
+ document is to provide a high-level explanation of the kernel
16
+ implementation step by step, aiding those who wish to learn about the
17
+ vLLM multi-head query attention kernel. After going through this
18
+ document, users will likely have a better understanding and feel easier
19
+ to follow the actual implementation.
20
+- Please note that this document may not cover all details, such as how
21
+ to calculate the correct index for the corresponding data or the dot
22
+ multiplication implementation. However, after reading this document
23
+ and becoming familiar with the high-level logic flow, it should be
24
+ easier for you to read the actual code and understand the details.
25
+
26
+Inputs
27
+------
28
+
29
+- The kernel function takes a list of arguments for the current thread
30
+ to perform its assigned work. The three most important arguments are
31
+ the input pointers ``q``, ``k_cache``, and ``v_cache``, which point
32
+ to query, key, and value data on global memory that need to be read
33
+ and processed. The output pointer ``out`` points to global memory
34
+ where the result should be written. These four pointers actually
35
+ refer to multi-dimensional arrays, but each thread only accesses the
36
+ portion of data assigned to it. I have omitted all other runtime
37
+ parameters here for simplicity.
38
+
39
+ .. code:: cpp
40
+
41
+ template<
42
+ typename scalar_t,
43
+ int HEAD_SIZE,
44
+ int BLOCK_SIZE,
45
+ int NUM_THREADS,
46
+ int PARTITION_SIZE = 0>
47
+ __device__ void paged_attention_kernel(
48
+ ... // Other side args.
49
+ const scalar_t* __restrict__ out, // [num_seqs, num_heads, max_num_partitions, head_size]
50
+ const scalar_t* __restrict__ q, // [num_seqs, num_heads, head_size]
51
+ const scalar_t* __restrict__ k_cache, // [num_blocks, num_kv_heads, head_size/x, block_size, x]
52
+ const scalar_t* __restrict__ v_cache, // [num_blocks, num_kv_heads, head_size, block_size]
53
+ ... // Other side args.
54
+ )
55
+
56
+- There are also a list of template arguments above the function
57
+ signature that are determined during compilation time. ``scalar_t``
58
+ represents the data type of the query, key, and value data elements,
59
+ such as FP16. ``HEAD_SIZE`` indicates the number of elements in each
60
+ head. ``BLOCK_SIZE`` refers to the number of tokens in each block.
61
+ ``NUM_THREADS`` denotes the number of threads in each thread block.
62
+ ``PARTITION_SIZE`` represents the number of tensor parallel GPUs (For
63
+ simplicity, we assume this is 0 and tensor parallel is disabled).
64
+- With these arguments, we need to perform a sequence of preparations.
65
+ This includes calculating the current head index, block index, and
66
+ other necessary variables. However, for now, we can ignore these
67
+ preparations and proceed directly to the actual calculations. It will
68
+ be easier to understand them once we grasp the entire flow.
69
+
70
+Concepts
71
+--------
72
+
73
+- Just before we dive into the calculation flow, I want to describe a
74
+ few concepts that are needed for later sections. However, you may
75
+ skip this section and return later if you encounter any confusing
76
+ terminologies.
77
+- **Sequence**: A sequence represents a client request. For example,
78
+ the data pointed to by ``q`` has a shape of
79
+ ``[num_seqs, num_heads, head_size]``. That represents there are total
80
+ ``num_seqs`` of query sequence data are pointed by ``q``. Since this
81
+ kernel is a single query attention kernel, each sequence only has one
82
+ query token. Hence, the ``num_seqs`` equals the total number of tokens
83
+ that are processed in the batch.
84
+- **Context**: The context consists of the generated tokens from the
85
+ sequence. For instance, ``["What", "is", "your"]`` are the context
86
+ tokens, and the input query token is ``"name"``. The model might
87
+ generate the token ``"?"``.
88
+- **Vec**: The vec is a list of elements that are fetched and
89
+ calculated together. For query and key data, the vec size
90
+ (``VEC_SIZE``) is determined so that each thread group can fetch and
91
+ calculate 16 bytes of data at a time. For value data, the vec size
92
+ (``V_VEC_SIZE``) is determined so that each thread can fetch and
93
+ calculate 16 bytes of data at a time. For example, if the
94
+ ``scalar_t`` is FP16 (2 bytes) and ``THREAD_GROUP_SIZE`` is 2, the
95
+ ``VEC_SIZE`` will be 4, while the ``V_VEC_SIZE`` will be 8.
96
+- **Thread group**: The thread group is a small group of
97
+ threads(\ ``THREAD_GROUP_SIZE``) that fetches and calculates one
98
+ query token and one key token at a time. Each thread handles only a
99
+ portion of the token data. The total number of elements processed by
100
+ one thread group is referred as ``x``. For example, if the thread
101
+ group contains 2 threads and the head size is 8, then thread 0
102
+ handles the query and key elements at index 0, 2, 4, 6, while thread
103
+ 1 handles the elements at index 1, 3, 5, 7.
104
+- **Block**: The key and value cache data in vLLM are split into
105
+ blocks. Each block stores data for a fixed number(\ ``BLOCK_SIZE``)
106
+ of tokens at one head. Each block may contain only a portion of the
107
+ whole context tokens. For example, if the block size is 16 and the
108
+ head size is 128, then for one head, one block can store 16 \* 128 =
109
+ 2048 elements.
110
+- **Warp**: A warp is a group of 32 threads(\ ``WARP_SIZE``) that
111
+ execute simultaneously on a stream multiprocessor (SM). In this
112
+ kernel, each warp processes the calculation between one query token
113
+ and key tokens of one entire block at a time (it may process multiple
114
+ blocks in multiple iterations). For example, if there are 4 warps and
115
+ 6 blocks for one context, the assignment would be like warp 0 handles
116
+ the 0th, 4th blocks, warp 1 handles the 1st, 5th blocks, warp 2
117
+ handles the 2nd block and warp 3 handles the 3rd block.
118
+- **Thread block**: A thread block is a group of
119
+ threads(\ ``NUM_THREADS``) that can access the same shared memory.
120
+ Each thread block contains multiple warps(\ ``NUM_WARPS``), and in
121
+ this kernel, each thread block processes the calculation between one
122
+ query token and key tokens of a whole context.
123
+- **Grid**: A grid is a collection of thread blocks and defines the
124
+ shape of the collection. In this kernel, the shape is
125
+ ``(num_heads, num_seqs, max_num_partitions)``. Therefore, each thread
126
+ block only handles the calculation for one head, one sequence, and
127
+ one partition.
128
+
129
+Query
130
+-----
131
+
132
+- This section will introduce how query data is stored in memory and
133
+ fetched by each thread. As mentioned above, each thread group fetches
134
+ one query token data, while each thread itself only handles a part of
135
+ one query token data. Within each warp, every thread group will fetch
136
+ the same query token data, but will multiply it with different key
137
+ token data.
138
+
139
+ .. code:: cpp
140
+
141
+ const scalar_t* q_ptr = q + seq_idx * q_stride + head_idx * HEAD_SIZE;
142
+
143
+ .. figure:: ../../assets/kernel/query.png
144
+ :alt: query
145
+ :width: 70%
146
+ :align: center
147
+
148
+ Query data of one token at one head
149
+
150
+- Each thread defines its own ``q_ptr`` which points to the assigned
151
+ query token data on global memory. For example, if ``VEC_SIZE`` is 4
152
+ and ``HEAD_SIZE`` is 128, the ``q_ptr`` points to data that contains
153
+ total of 128 elements divided into 128 / 4 = 32 vecs.
154
+
155
+ .. figure:: ../../assets/kernel/q_vecs.png
156
+ :alt: q_vecs
157
+ :width: 70%
158
+ :align: center
159
+
160
+ ``q_vecs`` for one thread group
161
+
162
+ .. code:: cpp
163
+
164
+ __shared__ Q_vec q_vecs[THREAD_GROUP_SIZE][NUM_VECS_PER_THREAD];
165
+
166
+- Next, we need to read the global memory data pointed to by ``q_ptr``
167
+ into shared memory as ``q_vecs``. It is important to note that each
168
+ vecs is assigned to a different row. For example, if the
169
+ ``THREAD_GROUP_SIZE`` is 2, thread 0 will handle the 0th row vecs,
170
+ while thread 1 handles the 1st row vecs. By reading the query data in
171
+ this way, neighboring threads like thread 0 and thread 1 can read
172
+ neighbor memory, achieving the memory coalescing to improve
173
+ performance.
174
+
175
+Key
176
+---
177
+
178
+- Similar to the "Query" section, this section introduces memory layout
179
+ and assignment for keys. While each thread group only handle one
180
+ query token one kernel run, it may handle multiple key tokens across
181
+ multiple iterations. Meanwhile, each warp will process multiple blocks
182
+ of key tokens in multiple iterations, ensuring that all context
183
+ tokens are processed by the entire thread group after the kernel run.
184
+ In this context, "handle" refers to performing the dot multiplication
185
+ between query data and key data.
186
+
187
+ .. code:: cpp
188
+
189
+ const scalar_t* k_ptr = k_cache + physical_block_number * kv_block_stride
190
+ + kv_head_idx * kv_head_stride
191
+ + physical_block_offset * x;
192
+
193
+- Unlike to ``q_ptr``, ``k_ptr`` in each thread will point to different
194
+ key token at different iterations. As shown above, that ``k_ptr``
195
+ points to key token data based on ``k_cache`` at assigned block,
196
+ assigned head and assigned token.
197
+
198
+ .. figure:: ../../assets/kernel/key.png
199
+ :alt: key
200
+ :width: 70%
201
+ :align: center
202
+
203
+ Key data of all context tokens at one head
204
+
205
+- The diagram above illustrates the memory layout for key data. It
206
+ assumes that the ``BLOCK_SIZE`` is 16, ``HEAD_SIZE`` is 128, ``x`` is
207
+ 8, ``THREAD_GROUP_SIZE`` is 2, and there are a total of 4 warps. Each
208
+ rectangle represents all the elements for one key token at one head,
209
+ which will be processed by one thread group. The left half shows the
210
+ total 16 blocks of key token data for warp 0, while the right half
211
+ represents the remaining key token data for other warps or
212
+ iterations. Inside each rectangle, there are a total 32 vecs (128
213
+ elements for one token) that will be processed by 2 threads (one
214
+ thread group) separately.
215
+
216
+ .. figure:: ../../assets/kernel/k_vecs.png
217
+ :alt: k_vecs
218
+ :width: 70%
219
+ :align: center
220
+
221
+ ``k_vecs`` for one thread
222
+
223
+ .. code:: cpp
224
+
225
+ K_vec k_vecs[NUM_VECS_PER_THREAD]
226
+
227
+- Next, we need to read the key token data from ``k_ptr`` and store
228
+ them on register memory as ``k_vecs``. We use register memory for
229
+ ``k_vecs`` because it will only be accessed by one thread once,
230
+ whereas ``q_vecs`` will be accessed by multiple threads multiple
231
+ times. Each ``k_vecs`` will contain multiple vectors for later
232
+ calculation. Each vec will be set at each inner iteration. The
233
+ assignment of vecs allows neighboring threads in a warp to read
234
+ neighboring memory together, which again promotes the memory
235
+ coalescing. For instance, thread 0 will read vec 0, while thread 1
236
+ will read vec 1. In the next inner loop, thread 0 will read vec 2,
237
+ while thread 1 will read vec 3, and so on.
238
+- You may still be a little confused about the overall flow. Don't
239
+ worry, please keep reading the next "QK" section. It will illustrate
240
+ the query and key calculation flow in a clearer and higher-level
241
+ manner.
242
+
243
+QK
244
+---
245
+
246
+- As shown the pseudo code below, before the entire for loop block, we
247
+ fetch the query data for one token and store it in ``q_vecs``. Then,
248
+ in the outer for loop, we iterate through different ``k_ptrs`` that
249
+ point to different tokens and prepare the ``k_vecs`` in the inner for
250
+ loop. Finally, we perform the dot multiplication between the
251
+ ``q_vecs`` and each ``k_vecs``.
252
+
253
+ .. code:: cpp
254
+
255
+ q_vecs = ...
256
+ for ... {
257
+ k_ptr = ...
258
+ for ... {
259
+ k_vecs[i] = ...
260
+ }
261
+ ...
262
+ float qk = scale * Qk_dot<scalar_t, THREAD_GROUP_SIZE>::dot(q_vecs[thread_group_offset], k_vecs);
263
+ }
264
+
265
+- As mentioned before, for each thread, it only fetches part of the
266
+ query and key token data at a time. However, there will be a cross
267
+ thread group reduction happen in the ``Qk_dot<>::dot`` . So ``qk``
268
+ returned here is not just between part of the query and key token dot
269
+ multiplication, but actually a full result between entire query and
270
+ key token data.
271
+- For example, if the value of ``HEAD_SIZE`` is 128 and
272
+ ``THREAD_GROUP_SIZE`` is 2, each thread's ``k_vecs`` will contain
273
+ total 64 elements. However, the returned ``qk`` is actually the
274
+ result of dot multiplication between 128 query elements and 128 key
275
+ elements. If you want to learn more about the details of the dot
276
+ multiplication and reduction, you may refer to the implementation of
277
+ ``Qk_dot<>::dot``. However, for the sake of simplicity, I will not
278
+ cover it in this document.
279
+
280
+Softmax
281
+-------
282
+
283
+- Next, we need to calculate the normalized softmax for all ``qk``\ s,
284
+ as shown above, where each :math:`x` represents a ``qk``. To do this,
285
+ we must obtain the reduced value of ``qk_max``\ (:math:`m(x)`) and
286
+ the ``exp_sum``\ (:math:`\ell(x)`) of all ``qk``\ s. The reduction
287
+ should be performed across the entire thread block, encompassing
288
+ results between the query token and all context key tokens.
289
+
290
+ .. math::
291
+ :nowrap:
292
+
293
+ \begin{gather*}
294
+ m(x):=\max _i \quad x_i \\ \quad f(x):=\left[\begin{array}{lll}e^{x_1-m(x)} & \ldots & e^{x_B-m(x)}\end{array}\right]\\ \quad \ell(x):=\sum_i f(x)_i \\
295
+ \quad \operatorname{softmax}(x):=\frac{f(x)}{\ell(x)}
296
+ \end{gather*}
297
+
298
+``qk_max`` and ``logits``
299
+~~~~~~~~~~~~~~~~~~~~~~~~~
300
+
301
+- Just right after we get the ``qk`` result, we can set the temporary
302
+ ``logits`` result with ``qk`` (In the end, the ``logits`` should
303
+ store the normalized softmax result). Also we can compare and collect
304
+ the ``qk_max`` for all ``qk``\ s that are calculated by current
305
+ thread group.
306
+
307
+ .. code:: cpp
308
+
309
+ if (thread_group_offset == 0) {
310
+ const bool mask = token_idx >= context_len;
311
+ logits[token_idx - start_token_idx] = mask ? 0.f : qk;
312
+ qk_max = mask ? qk_max : fmaxf(qk_max, qk);
313
+ }
314
+
315
+- Please note that the ``logits`` here is on shared memory, so each
316
+ thread group will set the fields for its own assigned context tokens.
317
+ Overall, the size of logits should be number of context tokens.
318
+
319
+ .. code:: cpp
320
+
321
+ for (int mask = WARP_SIZE / 2; mask >= THREAD_GROUP_SIZE; mask /= 2) {
322
+ qk_max = fmaxf(qk_max, VLLM_SHFL_XOR_SYNC(qk_max, mask));
323
+ }
324
+
325
+ if (lane == 0) {
326
+ red_smem[warp_idx] = qk_max;
327
+ }
328
+
329
+- Then we need to get the reduced ``qk_max`` across each warp. The main
330
+ idea is to make threads in warp to communicate with each other and
331
+ get the final max ``qk`` .
332
+
333
+ .. code:: cpp
334
+
335
+ for (int mask = NUM_WARPS / 2; mask >= 1; mask /= 2) {
336
+ qk_max = fmaxf(qk_max, VLLM_SHFL_XOR_SYNC(qk_max, mask));
337
+ }
338
+ qk_max = VLLM_SHFL_SYNC(qk_max, 0);
339
+
340
+- Finally, we can get the reduced ``qk_max`` from whole thread block by
341
+ compare the ``qk_max`` from all warps in this thread block. Then we
342
+ need to broadcast the final result to each thread.
343
+
344
+``exp_sum``
345
+~~~~~~~~~~~
346
+
347
+- Similar to ``qk_max``, we need to get the reduced sum value from the
348
+ entire thread block too.
349
+
350
+ .. code:: cpp
351
+
352
+ for (int i = thread_idx; i < num_tokens; i += NUM_THREADS) {
353
+ float val = __expf(logits[i] - qk_max);
354
+ logits[i] = val;
355
+ exp_sum += val;
356
+ }
357
+ ...
358
+ exp_sum = block_sum<NUM_WARPS>(&red_smem[NUM_WARPS], exp_sum);
359
+
360
+- Firstly, sum all exp values from each thread group, and meanwhile,
361
+ convert each entry of ``logits`` from ``qk`` to ``exp(qk - qk_max)``.
362
+ Please note, the ``qk_max`` here is already the max ``qk`` across the
363
+ whole thread block. And then we can do reduction for ``exp_sum``
364
+ across whole thread block just like the ``qk_max``.
365
+
366
+ .. code:: cpp
367
+
368
+ const float inv_sum = __fdividef(1.f, exp_sum + 1e-6f);
369
+ for (int i = thread_idx; i < num_tokens; i += NUM_THREADS) {
370
+ logits[i] *= inv_sum;
371
+ }
372
+
373
+- Finally, with the reduced ``qk_max`` and ``exp_sum``, we can obtain
374
+ the final normalized softmax result as ``logits``. This ``logits``
375
+ variable will be used for dot multiplication with the value data in
376
+ later steps. Now, it should store the normalized softmax result of
377
+ ``qk`` for all assigned context tokens.
378
+
379
+Value
380
+-----
381
+
382
+.. figure:: ../../assets/kernel/value.png
383
+ :alt: value
384
+ :width: 70%
385
+ :align: center
386
+
387
+ Value data of all context tokens at one head
388
+
389
+.. figure:: ../../assets/kernel/logits_vec.png
390
+ :alt: logits_vec
391
+ :width: 50%
392
+ :align: center
393
+
394
+ ``logits_vec`` for one thread
395
+
396
+.. figure:: ../../assets/kernel/v_vec.png
397
+ :alt: v_vec
398
+ :width: 70%
399
+ :align: center
400
+
401
+ List of ``v_vec`` for one thread
402
+
403
+- Now we need to retrieve the value data and perform dot multiplication
404
+ with ``logits``. Unlike query and key, there is no thread group
405
+ concept for value data. As shown in diagram, different from key token
406
+ memory layout, elements from the same column correspond to the same
407
+ value token. For one block of value data, there are ``HEAD_SIZE`` of
408
+ rows and ``BLOCK_SIZE`` of columns that are split into multiple
409
+ ``v_vecs``.
410
+- Each thread always fetches ``V_VEC_SIZE`` elements from the same
411
+ ``V_VEC_SIZE`` of tokens at a time. As a result, a single thread
412
+ retrieves multiple ``v_vec``\ s from different rows and the same
413
+ columns through multiple inner iterations. For each ``v_vec``, it
414
+ needs to be dot multiplied with the corresponding ``logits_vec``,
415
+ which is also ``V_VEC_SIZE`` elements from ``logits``. Overall, with
416
+ multiple inner iterations, each warp will process one block of value
417
+ tokens. And with multiple outer iterations, the whole context value
418
+ tokens are processd
419
+
420
+ .. code:: cpp
421
+
422
+ float accs[NUM_ROWS_PER_THREAD];
423
+ for ... { // Iteration over different blocks.
424
+ logits_vec = ...
425
+ for ... { // Iteration over different rows.
426
+ v_vec = ...
427
+ ...
428
+ accs[i] += dot(logits_vec, v_vec);
429
+ }
430
+ }
431
+
432
+- As shown in the above pseudo code, in the outer loop, similar to
433
+ ``k_ptr``, ``logits_vec`` iterates over different blocks and reads
434
+ ``V_VEC_SIZE`` elements from ``logits``. In the inner loop, each
435
+ thread reads ``V_VEC_SIZE`` elements from the same tokens as a
436
+ ``v_vec`` and performs dot multiplication. It is important to note
437
+ that in each inner iteration, the thread fetches different head
438
+ position elements for the same tokens. The dot result is then
439
+ accumulated in ``accs``. Therefore, each entry of ``accs`` is mapped
440
+ to a head position assigned to the current thread.
441
+- For example, if ``BLOCK_SIZE`` is 16 and ``V_VEC_SIZE`` is 8, each
442
+ thread fetches 8 value elements for 8 tokens at a time. Each element
443
+ is from different tokens at the same head position. If ``HEAD_SIZE``
444
+ is 128 and ``WARP_SIZE`` is 32, for each inner loop, a warp needs to
445
+ fetch ``WARP_SIZE * V_VEC_SIZE = 256`` elements. This means there are
446
+ a total of 128 \* 16 / 256 = 8 inner iterations for a warp to handle
447
+ a whole block of value tokens. And each ``accs`` in each thread
448
+ contains 8 elements that accumulated at 8 different head positions.
449
+ For the thread 0, the ``accs`` variable will have 8 elements, which
450
+ are 0th, 16th … 112th elements of a value head that are accumulated
451
+ from all assigned 8 tokens.
452
+
453
+LV
454
+---
455
+- Now, we need to perform reduction for ``accs`` within each warp. This
456
+ process allows each thread to accumulate the ``accs`` for the
457
+ assigned head positions of all tokens in one block.
458
+
459
+ .. code:: cpp
460
+
461
+ for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
462
+ float acc = accs[i];
463
+ for (int mask = NUM_V_VECS_PER_ROW / 2; mask >= 1; mask /= 2) {
464
+ acc += VLLM_SHFL_XOR_SYNC(acc, mask);
465
+ }
466
+ accs[i] = acc;
467
+ }
468
+
469
+- Next, we perform reduction for ``accs`` across all warps, allowing
470
+ each thread to have the accumulation of ``accs`` for the assigned
471
+ head positions of all context tokens. Please note that each ``accs``
472
+ in every thread only stores the accumulation for a portion of
473
+ elements of the entire head for all context tokens. However, overall,
474
+ all results for output have been calculated but are just stored in
475
+ different thread register memory.
476
+
477
+ .. code:: cpp
478
+
479
+ float* out_smem = reinterpret_cast<float*>(shared_mem);
480
+ for (int i = NUM_WARPS; i > 1; i /= 2) {
481
+ // Upper warps write to shared memory.
482
+ ...
483
+ float* dst = &out_smem[(warp_idx - mid) * HEAD_SIZE];
484
+ for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
485
+ ...
486
+ dst[row_idx] = accs[i];
487
+ }
488
+
489
+ // Lower warps update the output.
490
+ const float* src = &out_smem[warp_idx * HEAD_SIZE];
491
+ for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
492
+ ...
493
+ accs[i] += src[row_idx];
494
+ }
495
+
496
+ // Write out the accs.
497
+ }
498
+
499
+Output
500
+------
501
+
502
+- Now we can write all of calculated result from local register memory
503
+ to final output global memory.
504
+
505
+ .. code:: cpp
506
+
507
+ scalar_t* out_ptr = out + seq_idx * num_heads * max_num_partitions * HEAD_SIZE
508
+ + head_idx * max_num_partitions * HEAD_SIZE
509
+ + partition_idx * HEAD_SIZE;
510
+
511
+- First, we need to define the ``out_ptr`` variable, which points to
512
+ the start address of the assigned sequence and assigned head.
513
+
514
+ .. code:: cpp
515
+
516
+ for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) {
517
+ const int row_idx = lane / NUM_V_VECS_PER_ROW + i * NUM_ROWS_PER_ITER;
518
+ if (row_idx < HEAD_SIZE && lane % NUM_V_VECS_PER_ROW == 0) {
519
+ from_float(*(out_ptr + row_idx), accs[i]);
520
+ }
521
+ }
522
+
523
+- Finally, we need to iterate over different assigned head positions
524
+ and write out the corresponding accumulated result based on the
525
+ ``out_ptr``.
docs/source/index.rst
+1
@@ -98,6 +98,7 @@ Documentation
98
:caption: Developer Documentation
99
100
dev/engine/engine_index
101
+ dev/kernel/paged_attention
102
103
Indices and tables
104
==================