实践：about ollama

安装

curl -fsSL https://ollama.com/install.sh | sh

部署

ollama create example -f Modelfile

运行

ollama run example

终止(ollama加载的大模型将会停止占用显存，此时ollama属于失联状态，部署和运行操作失效，会报错：

Error: could not connect to ollama app, is it running?

需要启动后，才可以进行部署和运行操作)

systemctl stop ollama.service

终止后启动（启动后，可以接着使用ollama 部署和运行大模型）

systemctl start ollama.service

Modelfile contents:

FROM /home/wangbin/Desktop/Llama3/dir-unsloth.F16.gguf   PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>"  TEMPLATE """ <|im_start|>system {{ .System }}<|im_end|> <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """     PARAMETER temperature 0.8 PARAMETER num_ctx 8192  PARAMETER stop "<|system|>" PARAMETER stop "<|user|>" PARAMETER stop "<|assistant|>"    SYSTEM """You are a helpful, smart, kind, and efficient AI assistant.Your name is Aila. You always fulfill the user's requests to the best of your ability."""

ollama 参数：

(unsloth_env) wangbin@wangbin-LEGION-REN9000K-34IRZ:~/Desktop/Llama3$ ollama Usage:   ollama [flags]   ollama [command]  Available Commands:   serve       Start ollama   create      Create a model from a Modelfile   show        Show information for a model   run         Run a model   pull        Pull a model from a registry   push        Push a model to a registry   list        List models   ps          List running models   cp          Copy a model   rm          Remove a model   help        Help about any command  Flags:   -h, --help      help for ollama   -v, --version   Show version information  

卸载

1.Stop the Ollama Service     First things first, we need to stop the Ollama service from running. This ensures a smooth uninstallation process. Open your terminal and enter the following command:  sudo systemctl stop ollama  This command halts the Ollama service.     2.Disable the Ollama Service     Now that the service is stopped, we need to disable it so that it doesn’t start up again upon system reboot. Enter the following command:  sudo systemctl disable ollama  This ensures that Ollama won’t automatically start up in the future.     3.Remove the Service File     We need to tidy up by removing the service file associated with Ollama. Enter the following command:  sudo rm /etc/systemd/system/ollama.service  This deletes the service file from your system.     4.Delete the Ollama Binary     Next up, we’ll remove the Ollama binary itself. Enter the following command:  sudo rm $(which ollama)  This command removes the binary from your bin directory.     5.Remove Downloaded Models and Ollama User     Lastly, we’ll clean up any remaining bits and pieces. Enter the following commands one by one:  sudo rm -r /usr/share/ollama   sudo userdel ollama sudo groupdel ollama  These commands delete any downloaded models and remove the Ollama user and group from your system.

正文：

清洗PDF:

清洗PDF import PyPDF2 import re  def clean_extracted_text(text):     """Clean and preprocess extracted text."""     # Remove chapter titles and sections     text = re.sub(r'^(Introduction|Chapter \d+:|What is|Examples:|Chapter \d+)', '', text, flags=re.MULTILINE)     text = re.sub(r'ctitious', 'fictitious', text)     text = re.sub(r'ISBN[- ]13: \d{13}', '', text)     text = re.sub(r'ISBN[- ]10: \d{10}', '', text)     text = re.sub(r'Library of Congress Control Number : \d+', '', text)     text = re.sub(r'(\.|\?|\!)(\S)', r'\1 \2', text)  # Ensure space after punctuation     text = re.sub(r'All rights reserved|Copyright \d{4}', '', text)     text = re.sub(r'\n\s*\n', '\n', text)     text = re.sub(r'[^\x00-\x7F]+', ' ', text)     text = re.sub(r'\s{2,}', ' ', text)      # Remove all newlines and replace newlines only after periods     text = text.replace('\n', ' ')     text = re.sub(r'(\.)(\s)', r'\1\n', text)      return text  def extract_text_from_pdf(pdf_path):     """Extract text from a PDF file."""     with open(pdf_path, 'rb') as file:         reader = PyPDF2.PdfReader(file)         text = ''         for page in reader.pages:             if page.extract_text():                 text += page.extract_text() + ' '  # Append text of each page     return text  def main():     pdf_path = '/Users/charlesqin/Documents/The Art of Asking ChatGPT.pdf'  # Path to your PDF file     extracted_text = extract_text_from_pdf(pdf_path)     cleaned_text = clean_extracted_text(extracted_text)      # Output the cleaned text to a file     with open('cleaned_text_output.txt', 'w', encoding='utf-8') as file:         file.write(cleaned_text)  if __name__ == '__main__':     main()

微调代码：

from unsloth import FastLanguageModel import torch  from trl import SFTTrainer from transformers import TrainingArguments     max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.  # 4bit pre quantized models we support for 4x faster downloading + no OOMs. fourbit_models = [     "unsloth/mistral-7b-bnb-4bit",     "unsloth/mistral-7b-instruct-v0.2-bnb-4bit",     "unsloth/llama-2-7b-bnb-4bit",     "unsloth/gemma-7b-bnb-4bit",     "unsloth/gemma-7b-it-bnb-4bit", # Instruct version of Gemma 7b     "unsloth/gemma-2b-bnb-4bit",     "unsloth/gemma-2b-it-bnb-4bit", # Instruct version of Gemma 2b     "unsloth/llama-3-8b-bnb-4bit", # [NEW] 15 Trillion token Llama-3 ] # More models at https://huggingface.co/unsloth  model, tokenizer = FastLanguageModel.from_pretrained(     model_name = "unsloth/llama-3-8b-bnb-4bit",     max_seq_length = max_seq_length,     dtype = dtype,     load_in_4bit = load_in_4bit,     # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf )  model = FastLanguageModel.get_peft_model(     model,     r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128     target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",                       "gate_proj", "up_proj", "down_proj",],     lora_alpha = 16,     lora_dropout = 0, # Supports any, but = 0 is optimized     bias = "none",    # Supports any, but = "none" is optimized     # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!     use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context     random_state = 3407,     use_rslora = False,  # We support rank stabilized LoRA     loftq_config = None, # And LoftQ )  alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.  ### Instruction: {}  ### Input: {}  ### Response: {}"""  EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN def formatting_prompts_func(examples):     instructions = examples["instruction"]     inputs       = examples["input"]     outputs      = examples["output"]     texts = []     for instruction, input, output in zip(instructions, inputs, outputs):         # Must add EOS_TOKEN, otherwise your generation will go on forever!         text = alpaca_prompt.format(instruction, input, output) + EOS_TOKEN         texts.append(text)     return { "text" : texts, } pass  from datasets import load_dataset  file_path = "/home/Ubuntu/alpaca_gpt4_data_zh.json"   dataset = load_dataset("json", data_files={"train": file_path}, split="train")  dataset = dataset.map(formatting_prompts_func, batched = True,)     trainer = SFTTrainer(     model = model,     tokenizer = tokenizer,     train_dataset = dataset,     dataset_text_field = "text",     max_seq_length = max_seq_length,     dataset_num_proc = 2,     packing = False, # Can make training 5x faster for short sequences.     args = TrainingArguments(         per_device_train_batch_size = 2,         gradient_accumulation_steps = 4,         warmup_steps = 5,         max_steps = 60,         learning_rate = 2e-4,         fp16 = not torch.cuda.is_bf16_supported(),         bf16 = torch.cuda.is_bf16_supported(),         logging_steps = 1,         optim = "adamw_8bit",         weight_decay = 0.01,         lr_scheduler_type = "linear",         seed = 3407,         output_dir = "outputs",     ), )  trainer_stats = trainer.train()  model.save_pretrained_gguf("dir", tokenizer, quantization_method = "q4_k_m") model.save_pretrained_gguf("dir", tokenizer, quantization_method = "q8_0") model.save_pretrained_gguf("dir", tokenizer, quantization_method = "f16")

Ollama:

LM Studio:

我们使用经过Fine Tuning以后的Llama3大模型，询问它问题：

然后我们使用没有经过Fine Tuning的Llama3,还是用刚才的问题询问它：

Reference link：https://www.youtube.com/watch?v=oxTVzGwKeoU