# Weights & Biases sweep with multi gpu accelerate launch **URL:** \ **Category:** šŸ¤—Accelerate\ **Created:** [November 21, 2022, 3:01am UTC](https://discuss.huggingface.co/t/weights-biases-sweep-with-multi-gpu-accelerate-launch/26417 "2022-11-21T03:01:36Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![berkin](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@berkin](https://discuss.huggingface.co/u/berkin)\ **Post date:** [November 21, 2022, 3:01am UTC](https://discuss.huggingface.co/t/weights-biases-sweep-with-multi-gpu-accelerate-launch/26417/1 "2022-11-21T03:01:36Z")
Hi, I am trying to use Accelerate with multi-gpu on a single machine with a Weights and Biases sweep but I could not find any documentation specifically about this topic. I tried to accomplish this with the following approach but I am getting errors: In the main function, the accelerator is initialized as follows and the model parameters are taken from WANDB config. Then, the WAND config is setup and initialized as follows: ```auto def main() -> None: # Create accelerator for distributed training and logging accelerator = Accelerator( split_batches=True, mixed_precision="fp16", log_with="wandb", ) # Initialize wandb run if accelerator.is_main_process: accelerator.init_trackers( project_name=WANDB_PROJECT, init_kwargs={ "wandb": { "entity": WANDB_ENTITY, "dir": WANDB_EXPERIMENT_DIR, } }, ) # Log configuration from wandb tracker learning_rate = wandb.config.learning_rate base_model_name = wandb.config.base_model_name resize_resolution = wandb.config.resize_resolution model_type = MODEL_TYPES[base_model_name] if __name__ == " __main__": # Define sweep config SWEEP_CONFIG = { "method": "random", "early_terminate": { "type": "hyperband", "min_iter": 3, }, "name": "sweep", "metric": {"goal": "maximize", "name": "Valid Acc"}, "parameters": { # Log-uniform requires min/max values specified as base-e exponents "learning_rate": { "values": [ 1e-4, 1e-3, 1e-2, ] }, "base_model_name": { "values": [ "facebook/convnext-tiny-224", "facebook/convnext-small-224", "microsoft/swinv2-tiny-patch4-window8-256", ] }, "resize_resolution": {"values": ["model_base", "sd"]}, }, } # Initialize sweep by passing in config sweep_id = wandb.sweep( sweep=SWEEP_CONFIG, project=WANDB_PROJECT, entity=WANDB_ENTITY, ) # Start sweep job. wandb.agent(sweep_id, function=main, count=1) ``` Later, from the CLI I configure the accelerator parameters with ā€œaccelerate configā€ and run the following command: accelerate launch I am getting the following output and error: Create sweep with ID: z18ok842 Sweep URL: Create sweep with ID: 350at2y9 Sweep URL: Create sweep with ID: sca3d4qa Sweep URL: wandb: Agent Starting Run: 1oaggrmg with config: wandb: base\_model\_name: facebook/convnext-tiny-224 wandb: learning\_rate: 0.01 wandb: resize\_resolution: sd wandb: Agent Starting Run: y3o3i8tn with config: wandb: base\_model\_name: facebook/convnext-small-224 wandb: learning\_rate: 0.001 wandb: resize\_resolution: sd wandb: Agent Starting Run: vdcqv0d4 with config: wandb: base\_model\_name: facebook/convnext-small-224 wandb: learning\_rate: 0.01 wandb: resize\_resolution: model\_base Create sweep with ID: vp6bdpbb Sweep URL: wandb: Agent Starting Run: xw1qzfpw with config: wandb: base\_model\_name: facebook/convnext-tiny-224 wandb: learning\_rate: 0.01 wandb: resize\_resolution: model\_base Run xw1qzfpw errored: Error(ā€˜You must call wandb.init() before wandb.config.learning\_rateā€™) wandb: ERROR Run xw1qzfpw errored: Error(ā€˜You must call wandb.init() before wandb.config.learning\_rateā€™) Run y3o3i8tn errored: Error(ā€˜You must call wandb.init() before wandb.config.learning\_rateā€™) wandb: ERROR Run y3o3i8tn errored: Error(ā€˜You must call wandb.init() before wandb.config.learning\_rateā€™) Run vdcqv0d4 errored: Error(ā€˜You must call wandb.init() before wandb.config.learning\_rateā€™) wandb: ERROR Run vdcqv0d4 errored: Error(ā€˜You must call wandb.init() before wandb.config.learning\_rateā€™) Do you have a recommendation on how to integrate accelerate with wandb sweep? Thank you. ---
**Author:** ![morgan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/morgan/32/2293_2.png) [@morgan](https://discuss.huggingface.co/u/morgan)\ **Post date:** [November 23, 2022, 10:28am UTC](https://discuss.huggingface.co/t/weights-biases-sweep-with-multi-gpu-accelerate-launch/26417/2 "2022-11-23T10:28:01Z")
> [@berkin](#): > > ```auto > # Log configuration from wandb tracker > learning_rate = wandb.config.learning_rate > base_model_name = wandb.config.base_model_name > resize_resolution = wandb.config.resize_resolution > model_type = MODEL_TYPES[base_model_name] > > ``` Hey @berkin , I think this can be fixed by moving the following code under the test for whether its the main process, similar to how you have initialised wandb: ```auto if main_process: # Log configuration from wandb tracker learning_rate = wandb.config.learning_rate base_model_name = wandb.config.base_model_name resize_resolution = wandb.config.resize_resolution model_type = MODEL_TYPES[base_model_name] ``` ---
**Author:** ![berkin](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@berkin](https://discuss.huggingface.co/u/berkin)\ **Post date:** [November 23, 2022, 2:56pm UTC](https://discuss.huggingface.co/t/weights-biases-sweep-with-multi-gpu-accelerate-launch/26417/3 "2022-11-23T14:56:33Z")
> [@morgan](#): > > `main_process` Thank for your reply. I indeed made the modification you recommended but I still had to save the config to a json file so that all processes can access it: ``` if accelerator.is_main_process: shared_config = { "learning_rate": wandb.config.learning_rate, "base_model_name": wandb.config.base_model_name, "resize_resolution": wandb.config.resize_resolution, } # Share wandb.config from main process by writing it to disk json.dump(shared_config, open(SHARED_CONFIG_FILE, "w")) accelerator.wait_for_everyone() # Log configuration from wandb tracker shared_config = json.load(open(SHARED_CONFIG_FILE, "r")) learning_rate = shared_config["learning_rate"] base_model_name = shared_config["base_model_name"] resize_resolution = shared_config["resize_resolution"] ``` ---
**Author:** ![harshil-shah](https://avatars.discourse-cdn.com/v4/letter/h/7ba0ec/32.png) [@harshil-shah](https://discuss.huggingface.co/u/harshil-shah)\ **Post date:** [February 28, 2023, 4:48pm UTC](https://discuss.huggingface.co/t/weights-biases-sweep-with-multi-gpu-accelerate-launch/26417/4 "2023-02-28T16:48:54Z")
Hi, Iā€™m a bit confused about where to initialise the `Accelerator`: - If I initialise it inside the `main` function (the function that is passed to `wandb.agent`) then: - The first run in the sweep works, but for the second one I get the error: `AcceleratorState has already been initialized and cannot be changed, restart your runtime completely and pass mixed_precision='bf16' to Accelerate().` - Presumably this is due to calling `accelerator = Accelerator(log_with="wandb", mixed_precision="bf16")` again when the `main` function is run for a second time? Although @berkin this seems to be what youā€™re doing - did you see this error? - If I initialise it outside of the `main` function: - Again, the first run in the sweep works, but for the second one I get the error: `You can't use same Accelerator() instance with multiple models when using DeepSpeed.` - This occurs when calling `accelerator.prepare(...)` and seems to suggest that I should actually be creating a new `Accelerator` instance (i.e. by initialising again inside the `main` function). @berkin @morgan Iā€™d appreciate any suggestions; thanks! ---
**Author:** ![wise-east](https://avatars.discourse-cdn.com/v4/letter/w/8baadc/32.png) [@wise-east](https://discuss.huggingface.co/u/wise-east)\ **Post date:** [May 28, 2024, 6:57am UTC](https://discuss.huggingface.co/t/weights-biases-sweep-with-multi-gpu-accelerate-launch/26417/5 "2024-05-28T06:57:57Z")
@harshil-shah were you able to figure this out? @berkin @morgan Can you share the full script for our reference?