Skip to content

epl单机单卡和单机多卡训练step如何理解 #30

Open

Description

@SueeH

单机单卡:
启动命令:TF_CONFIG='{"cluster":{"worker":["127.0.0.1:49119"]},"task":{"type":"worker","index":0}}' CUDA_VISIBLE_DEVICES=0 bash ./scripts/train_dp.sh
image

单机双卡:
启动命令:TF_CONFIG='{"cluster":{"worker":["127.0.0.1:49119"]},"task":{"type":"worker","index":0}}' CUDA_VISIBLE_DEVICES=0,1 bash ./scripts/train_dp.sh
1693045873752

代码修改了一下:去掉了last_step限制,数据集repeat=10,将txt改为py,可执行。
resnet_dp.txt

想请教下,这个如何理解呢?每个卡分别跑了10step?

Activity

  1. changed the title [-]epl单卡显存降低和[/-] [+]epl单机单卡和单机多卡训练step如何理解[/+] on Sep 20, 2023
  2. adoda commented on Apr 24, 2024

    @adoda
    Collaborator

    现在配置的batch_size是batch_size/GPU,global_batch_size = batch_size*gpu_num
    数据量不变,增大gpu个数,一个epoch跑的step会线性减少。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions