Skip to main content

LLMs as Classifiers - Guide

1. Objective​

This guide provides step-by-step instructions on finetuning a model for Text Classification tasks on Emissary using our novel classification approach. In this approach, we add a classification head on top of the base LLMs that returns probabilities for a given set of labels. We recommend using Llama3.1-8B-instruct for this task.

2. Dataset Preparation​

Prepare your dataset in the appropriate format for the classification task.

Classification Data Format​

Each entry should contain:

  • Prompt: The input text for classification.
  • Completion: A binary value (0 or 1) indicating the classification output. Only 1 or 0 should be used to denote the classification result.

For Binary Classification Data (Single Label)​

JSONL Format

{
  "prompt": "This is a sample text for binary classification",
  "completion": 1
}

CSV Format

input,expected_output
"This is a sample text for binary classification",1
"This is another sample text for binary classification",0

For Multi-Class/Multi-Label Binary Classification​

In cases where multiple labels are evaluated independently, the completion should be a dictionary where each label is mapped to a binary value (0 or 1) to indicate its relevance.

JSONL Format

{
  "prompt": "This is a sample text for classification",
  "completion": {
    "Label_1": 1,
    "Label_2": 0,
    "Label_3": 1,
    "Label_N": 0
  }
}

CSV Format

input,expected_output
"This is a sample text for binary classification", "{"Label_1": 1, "Label_2": 0, "Label_3": 1, "Label_N": 0}"

3. Finetuning Preparation​

Please refer to the in-depth guide on Finetuning on Emissary here - Quickstart Guide.

Create Model Service​

Navigate to Dashboard arriving at Model Services, the default page on the Emissary platform.

  1. Click + NEW SERVICE in the dashboard. project_create_1
  2. In the pop-up, enter a new model service name, and click CREATE. project_create_2

Uploading Datasets​

A tile is created for your task. Click Manage to enter the task workspace. project_manage_1

  1. Click MANAGE in the Datasets Available tile. project_manage_1

  2. Click on + UPLOAD DATASET and select training and test datasets. dataset1

  3. Name datasets clearly to distinguish between training and test data (e.g., train_classification_data.csv, test_classification_data.csv).

4. Model Finetuning​

Now, go back one panel by clicking OVERVIEW and then click MANAGE in the Training Jobs tile. project_manage_1

Here, we’ll kick off finetuning. The shortest path to finetuning a model is by clicking + NEW TRAINING JOB, naming the output model, picking a backbone (base model), selecting the training dataset (you must have uploaded it in the step before), and finally hitting START NEW TRAINING JOB.

Selecting Classification Option​

When creating a new training job, you need to specify that you are performing a classification task to utilize the novel classification approach.

In the Training Job Creation page, locate the Task Type option. Select Classification from the given options.

This selection ensures that a classification head is added on top of the base LLM, enabling the model to return probabilities for the specified labels.

new_traning_job

A custom function that calculates a matching score for the given expected and predicted outputs, based on how well the binary expected labels match the predicted probabilities has been provided. Uncomment the suitable classification metric function to use it.

classification_metric

Training Parameter Configuration​

Please refer to the in-depth guide on configuring training parameters here - Finetuning Parameter Guide.

5. Model Monitoring & Evaluation​

Using Test Datasets​

Including a test dataset allows you to evaluate the model's performance during training.

  • Per Epoch Evaluation: The platform evaluates the model at each epoch using the test dataset.
  • Metrics and Outputs: View evaluation metrics and generated outputs for test samples.
  • Post completion of training, check scores in Training Job --> Artifacts.
    For the LLM model, expect the following:

evaluate_classification

6. Deployment​

Refer to the in-depth walkthrough on deploying a model on Emissary here - Deployment Guide.

Deploying your models allows you to serve them and integrate them into your applications.

Finetuned Model Deployment​

  1. Navigate to the Training Jobs Page. From the list of finetuning jobs, select the one you want to deploy. deployment_fine_tuned
  2. Go to the ARTIFACTS tab. artifacts1
  3. Select a Checkpoint to Deploy. checkpoint_evaluate
  4. Go to Deployments to check the status of you deployed model checkpoint_evaluate
  5. Once the model is deployed (as shown in the status), go to the testing tab. checkpoint_evaluate
  6. Test your samples in the the input box. checkpoint_evaluate

7. Best Practices​

  • Start Small: Begin with a smaller dataset to validate your setup.
  • Monitor Training: Keep an eye on training logs and metrics.
  • Iterative Testing: Use the test dataset to iteratively improve your model.
  • Data Format: Use the recommended data formats for your chosen model to ensure compatibility and optimal performance.