towhee.trainer.optimization.optimization.get_cosine_schedule_with_warmup¶
- towhee.trainer.optimization.optimization.get_cosine_schedule_with_warmup(optimizer: Optimizer, num_warmup_steps: int, num_training_steps: int, num_cycles: float = 0.5, last_epoch: int = -1)[source]¶
Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer.
- Parameters:
optimizer (
Optimizer) – The optimizer for which to schedule the learning rate.num_warmup_steps (
int) – The number of steps for the warmup phase.num_training_steps (
int) – The total number of training steps.num_cycles (
float, optional, defaults to 0.5) – The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine).last_epoch (
int, optional, defaults to -1) – The index of the last epoch when resuming training.
- Returns:
torch.optim.lr_scheduler.LambdaLRwith the appropriate schedule.
