towhee.models.perceiver.create_cross_attention.cross_attention_layer

towhee.models.perceiver.create_cross_attention.cross_attention_layer(num_q_channels: int, num_kv_channels: int, num_heads: int, dropout: float, activation_checkpoint: bool = False)[source]

Cross attention block for Perceiver https://arxiv.org/pdf/2103.03206.pdf.

Parameters:

  • num_q_channels (int) – Number of q channels.

  • num_kv_channels (int) – Number of k or v channels. k has the same channels as v.

  • num_heads (int) – Number of parallel attention heads.

  • dropout (nn.Module) – Dropout probability.

  • activation_checkpoint (bool) – Use activation checkpointing.

Return (nn.Module):

Configured cross attention layer.