正在写
感觉大模型炼丹（从零或微调）就像飞机驾驶员，一大堆超参数要调整。。数据也要各种处理，耗费大量精力，财力。
由于可能连不上HuggingFace，手动下载的模型要放到llama-factory根目录下，比如样例的meta-llama/Meta-Llama-3-8B-Instruct
https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/README_zh.md
LoRA 微调 （增量）预训练
llamafactory-cli train examples/train_lora/llama3_lora_pretrain.yaml
Every 1.0s: nvidia-smi ubuntu-cy: Sun Jul 7 21:01:46 2024
Sun Jul 7 21:01:46 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A |
| 30% 36C P2 107W / 350W | 3740MiB / 24576MiB | 3% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 Off | 00000000:02:00.0 Off | N/A |
| 30% 35C P2 113W / 350W | 3738MiB / 24576MiB | 3% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce RTX 3090 Off | 00000000:03:00.0 Off | N/A |
| 30% 41C P2 105W / 350W | 3738MiB / 24576MiB | 3% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA GeForce RTX 3090 Off | 00000000:04:00.0 Off | N/A |
| 30% 34C P2 108W / 350W | 3734MiB / 24576MiB | 3% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 57721 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 58786 C python 340MiB |
| 0 N/A N/A 1057988 C /home/linux/miniconda3/bin/python 3382MiB |
| 1 N/A N/A 57721 G /usr/lib/xorg/Xorg 4MiB |
| 1 N/A N/A 58786 C python 338MiB |
| 1 N/A N/A 1057989 C /home/linux/miniconda3/bin/python 3382MiB |
| 2 N/A N/A 57721 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 58786 C python 338MiB |
| 2 N/A N/A 1057990 C /home/linux/miniconda3/bin/python 3382MiB |
| 3 N/A N/A 57721 G /usr/lib/xorg/Xorg 4MiB |
预热：
随后正式开始：
Sun Jul 7 21:03:42 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A |
| 46% 56C P2 309W / 350W | 18928MiB / 24576MiB | 80% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 Off | 00000000:02:00.0 Off | N/A |
| 51% 59C P2 347W / 350W | 23776MiB / 24576MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce RTX 3090 Off | 00000000:03:00.0 Off | N/A |
| 58% 63C P2 272W / 350W | 23696MiB / 24576MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA GeForce RTX 3090 Off | 00000000:04:00.0 Off | N/A |
| 42% 54C P2 324W / 350W | 24142MiB / 24576MiB | 100% Default |
| | | N/A |
file added_tokens.json
[INFO|tokenization_utils_base.py:2159] 2024-07-07 21:01:21,299 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2159] 2024-07-07 21:01:21,299 >> loading file tokenizer_config.json
[WARNING|logging.py:313] 2024-07-07 21:01:21,826 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|loader.py:50] 2024-07-07 21:01:21,827 >> Loading dataset alpaca_en_demo.json...
07/07/2024 21:01:21 - WARNING - llamafactory.hparams.parser - ddp_find_unused_parameters needs to be set as False for LoRA in DDP training.
07/07/2024 21:01:21 - INFO - llamafactory.hparams.parser - Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
07/07/2024 21:01:21 - WARNING - transformers.tokenization_utils_base - Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/07/2024 21:01:21 - WARNING - llamafactory.hparams.parser - ddp_find_unused_parameters needs to be set as False for LoRA in DDP training.
07/07/2024 21:01:21 - INFO - llamafactory.hparams.parser - Process rank: 2, device: cuda:2, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
07/07/2024 21:01:22 - WARNING - transformers.tokenization_utils_base - Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/07/2024 21:01:22 - WARNING - transformers.tokenization_utils_base - Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
07/07/2024 21:01:23 - INFO - llamafactory.data.loader - Loading dataset alpaca_en_demo.json...
07/07/2024 21:01:23 - INFO - llamafactory.data.loader - Loading dataset alpaca_en_demo.json...
07/07/2024 21:01:23 - INFO - llamafactory.data.loader - Loading dataset alpaca_en_demo.json...
[INFO|configuration_utils.py:731] 2024-07-07 21:01:28,883 >> loading configuration file llama-3-8b/config.json
[INFO|configuration_utils.py:800] 2024-07-07 21:01:28,884 >> Model config LlamaConfig { "_name_or_path": "llama-3-8b", "architectures": [ "LlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "eos_token_id": 128001, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 14336, "max_position_embeddings": 8192, "mlp_bias": false, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 32, "num_key_value_heads": 8, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "rope_theta": 500000.0, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.42.3", "use_cache": true, "vocab_size": 128256 }
[INFO|modeling_utils.py:3553] 2024-07-07 21:01:29,021 >> loading weights file llama-3-8b/model.safetensors.index.json
[INFO|modeling_utils.py:1531] 2024-07-07 21:01:29,022 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1000] 2024-07-07 21:01:29,024 >> Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": 128001 }
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
07/07/2024 21:02:58 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
07/07/2024 21:02:58 - INFO - llamafactory.model.adapter - Fine-tuning method: LoRA
07/07/2024 21:02:58 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.misc - Found linear modules: up_proj,o_proj,down_proj,v_proj,q_proj,k_proj,gate_proj
07/07/2024 21:02:58 - INFO - llamafactory.model.adapter - Fine-tuning method: LoRA
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.misc - Found linear modules: gate_proj,q_proj,v_proj,down_proj,o_proj,up_proj,k_proj
[INFO|modeling_utils.py:4364] 2024-07-07 21:02:58,778 >> All model checkpoint weights were used when initializing LlamaForCausalLM.
[INFO|modeling_utils.py:4372] 2024-07-07 21:02:58,778 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at llama-3-8b. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training.
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
07/07/2024 21:02:58 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32.
07/07/2024 21:02:58 - INFO - llamafactory.model.adapter - Fine-tuning method: LoRA
07/07/2024 21:02:58 - INFO - llamafactory.model.model_utils.misc - Found linear modules: k_proj,v_proj,down_proj,up_proj,gate_proj,q_proj,o_proj
[INFO|configuration_utils.py:953] 2024-07-07 21:02:58,782 >> loading configuration file llama-3-8b/generation_config.json
[INFO|configuration_utils.py:1000] 2024-07-07 21:02:58,783 >> Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": 128001, "max_length": 4096, "temperature": 0.6, "top_p": 0.9 }
[INFO|checkpointing.py:103] 2024-07-07 21:02:58,793 >> Gradient checkpointing enabled.
[INFO|attention.py:80] 2024-07-07 21:02:58,793 >> Using torch SDPA for faster training and inference.
[INFO|adapter.py:302] 2024-07-07 21:02:58,793 >> Upcasting trainable params to float32.
[INFO|adapter.py:158] 2024-07-07 21:02:58,793 >> Fine-tuning method: LoRA
[INFO|misc.py:51] 2024-07-07 21:02:58,794 >> Found linear modules: q_proj,v_proj,down_proj,k_proj,o_proj,gate_proj,up_proj
07/07/2024 21:02:59 - INFO - llamafactory.model.loader - trainable params: 20,971,520 || all params: 8,051,232,768 || trainable%: 0.2605
07/07/2024 21:02:59 - INFO - llamafactory.model.loader - trainable params: 20,971,520 || all params: 8,051,232,768 || trainable%: 0.2605
[INFO|loader.py:196] 2024-07-07 21:02:59,635 >> trainable params: 20,971,520 || all params: 8,051,232,768 || trainable%: 0.2605
07/07/2024 21:02:59 - INFO - llamafactory.model.loader - trainable params: 20,971,520 || all params: 8,051,232,768 || trainable%: 0.2605
[INFO|trainer.py:642] 2024-07-07 21:02:59,655 >> Using auto half precision backend
[INFO|trainer.py:2128] 2024-07-07 21:03:03,203 >> ***** Running training *****
[INFO|trainer.py:2129] 2024-07-07 21:03:03,204 >> Num examples = 1,000
[INFO|trainer.py:2130] 2024-07-07 21:03:03,204 >> Num Epochs = 3
[INFO|trainer.py:2131] 2024-07-07 21:03:03,204 >> Instantaneous batch size per device = 2
[INFO|trainer.py:2134] 2024-07-07 21:03:03,204 >> Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:2135] 2024-07-07 21:03:03,204 >> Gradient Accumulation steps = 8
[INFO|trainer.py:2136] 2024-07-07 21:03:03,204 >> Total optimization steps = 45
[INFO|trainer.py:2137] 2024-07-07 21:03:03,211 >> Number of trainable parameters = 20,971,520
[INFO|callbacks.py:310] 2024-07-07 21:03:40,843 >> {'loss': 1.3376, 'learning_rate': 4.8492e-05, 'epoch': 0.32, 'throughput': 2093.78}
[INFO|callbacks.py:310] 2024-07-07 21:04:16,647 >> {'loss': 1.1486, 'learning_rate': 4.4151e-05, 'epoch': 0.64, 'throughput': 2210.62}
[INFO|callbacks.py:310] 2024-07-07 21:04:49,415 >> {'loss': 1.0795, 'learning_rate': 3.7500e-05, 'epoch': 0.96, 'throughput': 2205.54}
[INFO|callbacks.py:310] 2024-07-07 21:05:24,132 >> {'loss': 1.0536, 'learning_rate': 2.9341e-05, 'epoch': 1.28, 'throughput': 2198.01}
[INFO|callbacks.py:310] 2024-07-07 21:05:58,317 >> {'loss': 0.9959, 'learning_rate': 2.0659e-05, 'epoch': 1.60, 'throughput': 2195.79}
