towhee.trainer.optimization.optimization.get_cosine_with_hard_restarts_schedule_with_warmup

towhee.trainer.optimization.optimization.get_cosine_with_hard_restarts_schedule_with_warmup(optimizer: Optimizer, num_warmup_steps: int, num_training_steps: int, num_cycles: int = 1, last_epoch: int = -1)

Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer.

Parameters:

• optimizer (Optimizer) – The optimizer for which to schedule the learning rate.

• num_warmup_steps (int) – The number of steps for the warmup phase.

• num_training_steps (int) – The total number of training steps.

• num_cycles (int, optional, defaults to 1) – The number of hard restarts to use.

• last_epoch (int, optional, defaults to -1) – The index of the last epoch when resuming training.

Returns:

torch.optim.lr_scheduler.LambdaLR with the appropriate schedule.