Skip to content

Document flash attention contract #1068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
copybara-service wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Document flash attention contract #1068

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions gemma/flash_attention.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1100,11 +1100,12 @@ static HWY_INLINE void QDotKTilexUpTo8TransposedKDoubleWidthBF16(
// attention_shape - see struct definition for more details.
// att_cap - soft cap on attention logits
// att_out - MatPtrT<float> of shape (q_count, qkv_dim)
// exp_denominator_sums and max_logits: float* of shape:
// (RountedUpTo(q_count,4),)
// Need to be have multiple of 4 elements alocated and
// be initizalized If you need to compute over multiple chunks of kv's you can
// keep values between calls to this function and avoid explicit merge.
// exp_denominator_sums and max_logits - float* of shape (q_count,), holding
// the running online-softmax denominator and maximum logit per query. Only
// entries [0, q_count) are read and written. They are in-out: initialize them
// to 0.0f and -std::numeric_limits<float>::max() / 2 respectively before the
// first call. If you need to compute over multiple chunks of kvs, you can
// keep the values between calls to this function and avoid an explicit merge.
template <int kNumQueriesPerLoop, typename KV_T, typename Q_T, class DF,
class DU>
static HWY_INLINE void ComputeQKMacroTile(
Expand Down
Loading