VOOZH about

URL: https://huggingface.co/jukofyork/creative-writing-control-vectors-v3.0

⇱ jukofyork/creative-writing-control-vectors-v3.0 · Hugging Face

👁 image/png

This repo contains pre-generated control vectors in GGUF format for use with llama.cpp:

  • IMPORTANT: These new control vectors must use their respective de-bias control vector(s).
  • The code used to generate these can now be found at github.com/jukofyork/control-vectors.
  • All were generated with '--num_prompt_samples' set to the model's hidden state dimension.

Control vectors allow fine-tuned control over LLMs, enabling more precise/targeted text generation.

Table of Contents

Applying Control Vectors

To "de-bias" the model only:

Use the '--control-vector' option as follows:

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf

Alternatively for server mode:

llama-server --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf

This will apply the "language" de-bias control vector to the Mistral-Large-Instruct-2407 model.

You can apply multiple de-bias control vectors simultaneously like so:

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf \
 --control-vector mistral-large:123b-storytelling__debias.gguf \
 --control-vector mistral-large:123b-character_focus__debias.gguf

This will apply all 3 of the "writing style" de-bias control vectors.

To fully apply a positive or negative axis control vector with the default scale-factor:

Use the '--control-vector' option as follows:

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf \
 --control-vector mistral-large:123b-language__ornate.gguf

This will fully apply (ie: with a scale-factor of 1.0) the (positive-axis) "ornate language" control vector.

IMPORTANT: The positive and negative axis control vectors must be used along with the relevant de-bias control vector - they cannot be used on their own!

You can fully apply multiple positive or negative axis control vectors like so:

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf \
 --control-vector mistral-large:123b-language__ornate.gguf \
 --control-vector mistral-large:123b-storytelling__debias.gguf \
 --control-vector mistral-large:123b-storytelling__descriptive.gguf \
 --control-vector mistral-large:123b-character_focus__debias.gguf \
 --control-vector mistral-large:123b-character_focus__dialogue.gguf

This will fully apply (ie: with a scale-factor of 1.0) all 3 of the (positive-axis) "writing style" control vectors.

NOTE: Fully applying too many positive or negative axis control vector simultaneously may damage the model's output.

To partially apply a positive or negative axis control vector using a custom scale-factor: 

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf \
 --control-vector-scaled mistral-large:123b-language__ornate.gguf 0.5

This will partially apply the (positive-axis) "ornate language" control vector with a scale-factor of 0.5 (ie: half the full effect).

IMPORTANT: The positive and negative axis control vectors must be used along with the relevant de-bias control vector - they cannot be used on their own!

You can partially apply multiple positive or negative axis control vectors like so:

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf \
 --control-vector-scaled mistral-large:123b-language__ornate.gguf 0.5 \
 --control-vector mistral-large:123b-storytelling__debias.gguf \
 --control-vector-scaled mistral-large:123b-storytelling__descriptive.gguf 0.3 \
 --control-vector mistral-large:123b-character_focus__debias.gguf \
 --control-vector-scaled mistral-large:123b-character_focus__dialogue.gguf 0.2

This will partially apply all 3 of the (positive-axis) "writing style" control vectors with varying weights.

The theoretical upper bound value for equal weights is between 1/n and sqrt(1/n) depending on how correlated the n control vector directions are, eg:

  • For n = 1 use the default scale-factor of 1.0 for comparison with the values below.
  • For n = 2 is between 1/2 ≈ 0.5 and sqrt(1/2) ≈ 0.707.
  • For n = 3 is between 1/3 ≈ 0.333 and sqrt(1/3) ≈ 0.577.
  • For n = 4 is between 1/4 ≈ 0.25 and sqrt(1/4) ≈ 0.5.
  • For n = 5 is between 1/5 ≈ 0.2 and sqrt(1/5) ≈ 0.447.

and so on.

The way the positive and negative axis control vectors are calibrated means you can negate the scale-factors too, eg:

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf \
 --control-vector-scaled mistral-large:123b-language__ornate.gguf -0.5

is equivalent to:

llama-cli --model <model>.gguf [other CLI arguments] \
 --control-vector mistral-large:123b-language__debias.gguf \
 --control-vector-scaled mistral-large:123b-language__simple.gguf 0.5

NOTE: It is possible to use scale-factors greater than 1.0, but if too large it will eventually damage the model's output.

Important Notes

  1. Always include the relevant "de-bias" control vector as well as the positive-axis/negative-axis control vector - they cannot be used on their own!
  2. Do not mix both sides of a positive/negative axis at the same time (eg: '--control-vector language__simple.gguf' and '--control-vector language__ornate.gguf' will just cancel out and have no effect...).
  3. Ensure your llama.cpp version is up to date (multi-vector support added 27/06/24 in #8137).

Command Line Generator

Courtesy of gghfez, a utility to easily generate command line options for llama.cpp:

👁 image/png

You can run this tool directly on GitHub Pages.

Direct Links

Very Large Models

Large Models

Medium Models

Small Models

Algorithm Details

1. First we create a set of pre/post "prompt stems":

The Cartesian product of these gives us 2500 (ie: 50 x 50) different "You are an author" type sentences.

2. Then we create several different creative-writing axis "continuations":

A set of 3 different "writing style" axis:

The 4 elements of the Dark Tetrad:

An "Optimism vs Nihilism" axis to compliment the Dark Tetrad axis:

3. Then we collect a large number of creative-writing prompts:

  • I used Sao10K/Short-Storygen-v2 and a couple of other sources to get 11835 creative-writing prompts in total (see the 'writing_prompts.txt' file).
  • The jq command is very useful for extracting the prompts only from these datasets.

4. Run the model on a random sample of (prompt-stem, continuation, creative-writing prompts) combinations:

The Cartesian product of: 2500 prompt-stem sentences x 10 continuation sentences x 11835 story prompts ≈ 300M possible combinations.

  • It is important that the same prompt-stem sample sentence be used with each ("baseline", "negative", "positive") triplet.
  • It is also important that the same (prompt-stem, continuation) sample sentence be used with the"negative" and "positive" members of the same triplet.
  • The suggested value of "hidden_size" for the --num_prompt_samples option is because the theory regarding estimation of covariance matrices shows we need at the very least a minimum of one sample per feature (this may be overkill due to us only retaining the top Eigenvectors though...).

5. Create a pair of "differenced datasets" by subtracting the corresponding "baseline" class's sample from both of the other 2 classes' samples:

  • The reason for this is so that we "centre" the data around the "baseline" (i.e., set the "baseline" as the origin and look for vector directions that point away from it).
  • This is in contrast to assuming the difference of the means is the "centre" for a 2-class version of this using PCA on the covariance matrix of the differences (i.e., the "standard" method of creating control vectors).

6. Now we take our two "differenced datasets" held in data matrices A and B (with rows as samples and columns as features):

  1. Create the cross-covariance matrix, C = A^T * B.
  2. Next we symmetrise, C' = (C^T + C) / 2.
  3. Perform an eigendecomposition, C' = Q * Λ * Q^(-1).
  4. Since we symmetrised the matrix, the eigenvectors (Q) and eigenvalues (Λ) will all be real-valued.
  5. Arrange the eigenvectors in descending order based on their corresponding eigenvalues.
  6. Once the eigenvectors are sorted, discard the eigenvalues as they won't be needed again.

The reason for using the cross-covariance matrix instead of the covariance matrix:

  • The covariance matrix of a differenced dataset exemplifies directions in A or B (ie: think about the expansion of (a-b)² = a² + b² -2×a×b).
  • The cross-covariance matrix of a differenced dataset exemplifies directions in A and B (ie: akin to a×b, with no a² or b² terms).

The reason for creating the symmetrised matrix is two-fold:

  • To avoid complex-valued eigenvectors that tell us about rotations (which we can't actually make use of here anyway).
  • To specifically try to find opposing/balanced "axis" for our different traits (i.e., we don't want to find positively correlated directions nor unbalanced directions).

7. So now we have a set of "directions" to examine:

  • It turns out that 90% of the time the principal eigenvector (i.e., the eigenvector with the largest corresponding eigenvalue) is the one you want.
  • In the ~10% of cases where it is not the principal eigenvector or split between a couple of different eigenvectors, we (greedily) create a "compound direction" by examining the discriminant ratio of each direction.

8. Finally, we project the "direction" to reorient and scale as necessary:

  • There is no reason the eigenvectors point in the direction we want, so 50% of the time we have to flip all the signs by projecting our (differenced) "desired" dataset on to the (unit norm) direction and then test the sign of the mean.
  • Due to the way the LLMs work via the "residual stream", the hidden states tend to get larger and larger as the layers progress, so to normalize this we also scale by the magnitude of the mean of the same projection as above.
  • To better separate the "bias" effect from the positive/negative axis (and to make the positive/negative end equidistant from the model's "baseline" behaviour) we store the mid point of these means in the de-bias control vector and then subtract the midpoint from both the positive and negative axis' control vectors.

NOTES:

  • I have found the above can be applied to every layer, but often the last layer will have hidden state means that are 10-100x larger than the rest, so I have excluded these from all I have uploaded here.
  • I have tried many other different eigendecompositions: PCA on the 2-class differenced datasets, PCA on the joined 2-class/3-class datasets, solving generalized eigensystems similar to CCA, and so on.
  • The "balanced" directions / "axis" this method finds are the exact opposite of those needed for the Refusal in LLMs is mediated by a single direction paper.

Changelog

Downloads last month
32,365
GGUF
Model size
319k params
Architecture
controlvector
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including jukofyork/creative-writing-control-vectors-v3.0