gabrielkyebambo/Llama-3.2-3B-Instruct-TaskPlans

Model Card for Model ID

This model is a transformer-based neural network designed to generate safe and efficient task plans for robotic operations. It translates human instructions into detailed task plans in JSON format, ensuring that tasks are executed safely and efficiently. The model incorporates spatial relationships, safety constraints, and performance metrics to provide comprehensive and contextually relevant task plans for robotic systems.

Model Details

Model Description

This model is a transformer-based neural network specifically designed to generate safe and efficient task plans for robotic operations. It takes human instructions as input and outputs detailed task plans in JSON format, which include spatial relationships, task planning steps, and performance metrics. The model is particularly useful for applications where precise and safe robotic manipulation is required, such as in manufacturing, service robotics, and automation.

• Developed by: Shaanxi University of Science and Technology/Fehead
• Funded by [optional]: Shaanxi Provincial Department of Education Service Local Special Plan Project (22JC019)
• Shared by [optional]: Shaanxi University of Science and Technology/Fehead
• Model type: Transformer
• Language(s) (NLP): English
• License: [More Information Needed]
• Finetuned from model [optional]: meta-llama/Llama-3.2-3B-Instruct

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

The model can be used directly for generating task plans for robotic operations based on human instructions. This is particularly useful in scenarios where precise and safe robotic manipulation is required. The model takes a human instruction as input and outputs a detailed task plan in JSON format, which includes spatial relationships, task planning steps, and performance metrics. This makes it suitable for applications in: Manufacturing: Automating repetitive tasks and improving efficiency. Service Robotics: Assisting in tasks such as cleaning, object retrieval, and placement. Automation: Enhancing the capabilities of automated systems in various industries.

Downstream Use [optional]

The model can be fine-tuned or integrated into larger systems for more specialized applications. For example: Custom Task Planning: Fine-tuning the model on specific tasks or environments to improve accuracy and relevance. Robotic Control Systems: Integrating the model into robotic control systems to provide real-time task planning and execution. Human-Robot Interaction: Enhancing interaction systems where robots need to understand and execute human instructions accurately.

Out-of-Scope Use

While the model is designed for generating safe and efficient task plans, it is important to note its limitations and potential misuse: Misuse: The model should not be used in scenarios where the safety constraints are not applicable or where the environment is significantly different from the training data. Limitations: The model's performance may degrade in environments with high variability or in tasks that require highly specialized knowledge not covered in the training data. Ethical Considerations: Users should ensure that the model is used ethically and responsibly, avoiding any scenarios that could lead to harm or unethical behavior.

[More Information Needed]

Bias, Risks, and Limitations

[More Information Needed]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

To get started with the model, you can use the following code to load the model, generate task plans based on human instructions, and parse the output. This example demonstrates how to run the fine-tuned model locally. Prerequisites Ensure you have the necessary dependencies installed. You can install them using pip:

pip install transformers torch

Below is an example of how to run the fine-tuned model locally to generate task plans based on human instructions:

import os import json import torch from textwrap import dedent from typing import Dict, List, Optional, Union, Any from functools import lru_cache

from transformers import ( AutoModelForCausalLM, AutoTokenizer, pipeline, Pipeline )

class ModelManager: def init(self, model_name: str): self.model_name = model_name self.tokenizer = None self.model = None self.pipe = None

@lru_cache(maxsize=1)
def get_tokenizer(self):
    """Load tokenizer if not already loaded."""
    if self.tokenizer is None:
        self.tokenizer = AutoTokenizer.from_pretrained(self.model_name)
    return self.tokenizer

def load_model(self):
    """Load the model with proper error handling and performance optimizations."""
    try:
        if self.model is None:
            self.model = AutoModelForCausalLM.from_pretrained(
                self.model_name,
                torch_dtype=torch.float16,
                device_map="auto",
                # Add performance optimizations
                low_cpu_mem_usage=True,
                offload_folder="offload",
            )
            
            tokenizer = self.get_tokenizer()
            current_embed_size = self.model.get_input_embeddings().weight.size(0)
            if len(tokenizer) != current_embed_size:
                self.model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=8)
            
        return self.model
    except Exception as e:
        raise RuntimeError(f"Failed to load model: {str(e)}") from e

def get_pipeline(self) -> Pipeline:
    """Get or create the generation pipeline with optimized settings."""
    if self.pipe is None:
        model = self.load_model()
        tokenizer = self.get_tokenizer()
        self.pipe = pipeline(
            task="text-generation",
            model=model,
            tokenizer=tokenizer,
            max_new_tokens=1000,  # Increased for longer responses
            return_full_text=False,
            # Add performance settings
            batch_size=1,
            num_workers=4,
        )
    return self.pipe

def create_prompt(self, user_input: str) -> str:
    """Create a formatted prompt for the model."""
    prompt = dedent(f"""
    Instruction:
    {user_input}
    
    Please provide the response in a valid JSON format with spatial_relationships, task_plan, and performance_metrics.
    """)

    messages = [
        {
            "role": "system",
            "content": "Generate a task plan in JSON format with spatial relationships, task planning, and performance metrics.",
        },
        {"role": "user", "content": prompt},
    ]
    
    return self.get_tokenizer().apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

def parse_json_response(self, text: str) -> Dict[str, Any]:
    """Parse the model output into a JSON object."""
    try:
        # Clean up the text to extract just the JSON part
        # Find the first '{' and last '}'
        start_idx = text.find('{')
        end_idx = text.rfind('}') + 1
        
        if start_idx == -1 or end_idx == 0:
            raise ValueError("No JSON object found in response")
            
        json_str = text[start_idx:end_idx]
        
        # Parse the JSON
        return json.loads(json_str)
    except json.JSONDecodeError as e:
        raise ValueError(f"Failed to parse JSON: {str(e)}\nOriginal text: {text}")

def generate(self, prompt: str) -> Dict[str, Any]:
    """Generate and parse response for a given prompt."""
    try:
        pipe = self.get_pipeline()
        outputs = pipe(prompt)
        raw_response = outputs[0]["generated_text"]
        # return self.parse_json_response(raw_response)
        return raw_response
    except Exception as e:
        raise RuntimeError(f"Generation failed: {str(e)}") from e

def save_response(response: Dict[str, Any], filename: str = "response.json"): """Save the response to a JSON file.""" with open(filename, 'w') as f: json.dump(response, f, indent=2)

def main(): model_name = "gabrielkyebambo/Llama-3.2-3B-Instruct-TaskPlans" manager = ModelManager(model_name)

try:
    while True:
        user_input = input('Enter prompt (or "quit" to exit): ')
        if user_input.lower() == 'quit':
            break
            
        prompt = manager.create_prompt(user_input)
        print(f'\nFormatted prompt:\n{prompt}\n')
        
        response = manager.generate(prompt)
        
        # Save response to file
        save_response(response)
        
        # Pretty print the response
        print("\nGenerated task plan (formatted):")
        print(json.dumps(response, indent=2))
        
except KeyboardInterrupt:
    print("\nExiting gracefully...")
except Exception as e:
    print(f"An error occurred: {str(e)}")

if name == "main": main()

Training Details

Training Data

The model was trained on a dataset consisting of 28,715 human instructions paired with corresponding task plans. The dataset is divided into training, validation, and test sets as follows: Training set size: 22,972 examples Validation set size: 4,594 examples Test set size: 1,149 examples

Each example in the dataset includes the following features: instruction: Human instruction in natural language. task_plan: Corresponding task plan in JSON format. text: Combined text of instruction and task plan. token_count: Number of tokens in the text.

Training Procedure

Preprocessing Tokenization: Human instructions and task plans were tokenized using the HF中国镜像站 tokenizer. Normalization: Text data was normalized to ensure consistency. Augmentation: Additional synthetic data was generated to cover edge cases and improve robustness.

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

• Training regime:
• fp16 mixed precision Batch size: 4 (per device) Evaluation batch size: 1 (per device) Gradient accumulation steps: 2 Optimizer: paged_adamw_8bit Learning rate: 3e-4 Number of epochs: 3 Max sequence length: 1024 Evaluation strategy: steps Evaluation steps: 0.2 (20% of training steps) Save steps: 0.2 (20% of training steps) Logging steps: 10 Warmup ratio: 0.1 Save total limit: 2 Learning rate scheduler: cosine Reporting: tensorboard Save safetensors: True Max gradient norm: 0.5 Dataset kwargs: add_special_tokens: False append_concat_token: False

Speeds, Sizes, Times [optional]

Training time: 27 hours Hardware used: Single GPU (4060 TI 16GB) Checkpoint size: [Specify Checkpoint Size] Total parameters: 3,261,402,112 Trainable parameters: 48,627,712 (1.4910% of total parameters)

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Testing Data The model was evaluated on a separate test dataset consisting of 1,149 human instructions and corresponding task plans.

Energy consumption: [Specify Average Energy Consumption]

[More Information Needed]

Factors

[More Information Needed]

Metrics

Metrics Accuracy: 98 Safety Compliance: 95 Efficiency Metrics: 97

[More Information Needed]

Results

Human Instruction: "go to the dining room, pick up the mug from the dining table, then take it to the kitchen and put it inside the sink"

Formatted prompt: <|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023 Today Date: 02 Feb 2025

Generate a task plan in JSON format with spatial relationships, task planning, and performance metrics.<|eot_id|><|start_header_id|>user<|end_header_id|>

Instruction: go to the dinning room, pick up the mug from the dinning table, then take it to the kitchen and put it inside the sink

Please provide the response in a valid JSON format with spatial_relationships, task_plan, and performance_metrics.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{ "spatial_relationships": { "object": "mug", "source": { "location": "dining room", "container_or_surface": "dining table", "height": "0.05m", "weight_range": [0.2, 0.3], "grip_type": "power" }, "destination": { "location": "kitchen", "container_or_surface": "sink", "height": "0.95m", "weight_range": [0.1, 0.2], "grip_type": "power" }, "navigation_constraints": { "min_obstacle_distance": "0.7m", "max_velocity": "1.0m/s", "max_acceleration": "1.0m/s²" }, "task_constraints": { "success_criteria": [ "navigation_completed_safely", "object_grasped_securely", "object_transported_safely", "object_placed_correctly" ], "safety_constraints": { "collision_avoidance": "active", "force_limits": { "grasp_force": "24.08N", "transport_force": "17.17N", "placement_force": "2.45N" }, "height_constraints": { "min_height": "0.05m", "max_height": "0.95m" }, "orientation_constraints": { "object_stability": "high", "table_surface_stability": "high" } } }, "performance_metrics": { "success_criteria": [ "navigation_completed_safely", "object_grasped_securely", "object_transported_safely", "object_placed_correctly" ], "safety_metrics": { "collision_avoidance": "verified", "force_limits_maintained": "verified", "height_constraints_maintained": "verified", "orientation_constraints_maintained": "verified" }, "estimated_metrics": { "total_distance": "13.6m", "estimated_time": "59s", "energy_consumption": "18.3Wh" }, "critical_monitoring": [ "object_tracking", "force_feedback", "obstacle_detection" ] } } }

[More Information Needed]

Summary

This model provides a robust solution for generating task plans for robotic operations, ensuring safety and efficiency. It is suitable for applications in manufacturing, service robotics, and automation. For more detailed information, refer to the full model card. The dataset for the model is currently being expanded to include a broader range of human instructions for robotic operations. This expansion aims to enhance the model's ability to handle more diverse and complex tasks in human-robot collaboration scenarios. The new dataset will cover various manufacturing tasks, including assembly, disassembly, and maintenance operations, to improve the model's generalization and adaptability

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA 4060 TI 16GB GPU.
• Hours used: > 27 hours
• Cloud Provider: Trained locally
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

BibTeX:

@article{qi2024safety, title={Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs}, author={Qi, Yong and Kyebambo, Gabriel and Xie, Siyuan and Shen, Wei and Wang, Shenghui and Xie, Bitao and He, Bin and Wang, Zhipeng and Jiang, Shuo}, journal={arXiv preprint arXiv:2405.17846}, year={2024} }

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[email protected]