embeddings_livnlearnversion

The purpose of this notebook is to study and evaluate the use of an embedding model in user segmentation

Task¶

User segmentation / profiling (without training)

Data¶

lets say if i have data for what locations people are interested in (maybe they’ve searched those or visited those)
i’ve formatted the data like:
“{user1}”: {
“{name of place} {city} {state} {country}”: {frequency of search/visit},
“{name of place} {city} {state} {country}”: {frequency of search/visit},
…
}

Plan¶

we’ve all spent time doing strenous feature selection and engineering using distribution tests, predictive powers, iterations etc etc to select key features, encode them with things like OHE and train models creating user vectors and clustering

but what if instead we could just get raw data encoded directly and user vectors created with context of each feature making user itself queryable- no predefined feature engg, vectorization or clustering – we use the power of pre trained embedding models to bring in the knowledge of context

Here’s how the process could look like:

get distinct strings (lets call them domain entries)
get embedding for each distinct domain entry – {name of place} {city} {state} {country} – using openai embedding model
weigh each embedding by frequency and average them together to create a user vector
encode query vector using same embedding model
list all users similar to a description (using a cosine similarity for instance)

Purpose of this notebook¶

To test the embedding model and get more insights on how they encode – what can we do with their embeddings, characterstics of embedding spaces – how does doing a weighted average look like in principle

concerns in real life behaviour could have tens of thousands of distinct placecodes for a user, combining them could lead to things like

dilution by too frequent behaviours, too many domains
dilution of weaker traits by combination
information loss
missclassification etc. etc. I discuss them below in the Test section

We use Open AI text embedding 3 small to inspect and play with the embeddings
To begin with, the import section below imports the necessary libraries and sets up the embedding function

Imports¶

In [50]:

from dotenv import load_dotenv
import os

import pandas as pd
import numpy as np

from sklearn.metrics.pairwise import cosine_similarity
from sklearn.cluster import KMeans

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

from matplotlib import pyplot as plt
import seaborn as sns
import plotly.express as px
import plotly.io as pio
pio.renderers.default = "notebook"

In [51]:

import openai
from openai import OpenAI

load_dotenv()
openai_api_key = os.getenv("OPENAI_API_KEY")

openai_client = OpenAI(
    api_key=os.getenv("OPENAI_API_KEY")
)
def get_embedding(text, deployment_name="text-embedding-3-small"):
    response = openai_client.embeddings.create(
        input=[text],
        model=deployment_name
    )
    return response.data[0].embedding

Example 1¶

We start with the standard example they quote everywhere – basically if the context makes sense and its the same vector space, then king – man + woman -> queen

In [52]:

texts = {
    "king": "king",
    "man": "man",
    "woman": "woman",
    "queen": "queen"
}

In [53]:

embeddings = {k: np.array(get_embedding(v)) for k, v in texts.items()}

In [54]:

king_vec = embeddings["king"]
man_vec = embeddings["man"]
woman_vec = embeddings["woman"]
queen_vec = embeddings["queen"]

In [55]:

analogy_vec = king_vec - man_vec + woman_vec

In [56]:

similarity = cosine_similarity([analogy_vec], [queen_vec])[0][0]
print(f"Cosine similarity with 'queen': {similarity:.4f}")

Cosine similarity with 'queen': 0.6162

In [57]:

print("Similarity of analogy vector to all texts:")
for word, vec in embeddings.items():
    sim = cosine_similarity([analogy_vec], [vec])[0][0]
    print(f"{word}: {sim:.4f}")

Similarity of analogy vector to all texts:
king: 0.7637
man: 0.1922
woman: 0.6023
queen: 0.6162

Even though this is quite similar to queen (& woman) – its still most similar to king

In [58]:

embeddings['analogy'] = analogy_vec
labels = list(embeddings.keys())
embedding_matrix = np.array(list(embeddings.values()))

you could choose either PCA or t-SNE to visualize both will allow to reduce this 1536 dimension to a 2-3 dim one for visualization ( its important to note they’re approximations and both of them work differently – inspite of what people keep saying :/ ) – in my experience tSNE better captures this higher dimensionality but you’re free to try both – this case is very simple so it doesn’t matter but you can add more examples and see

In [59]:

pca_3d = PCA(n_components=3)
pca_result = pca_3d.fit_transform(embedding_matrix)

fig_pca = px.scatter_3d(
    x=pca_result[:, 0], y=pca_result[:, 1], z=pca_result[:, 2],
    text=labels,
    title="3D PCA of OpenAI Embeddings"
)
fig_pca.show()

In [60]:

# tsne_3d = TSNE(n_components=3, perplexity=3, n_iter=1000, random_state=42)
# tsne_result = tsne_3d.fit_transform(embedding_matrix)

# fig_tsne = px.scatter_3d(
#     x=tsne_result[:, 0], y=tsne_result[:, 1], z=tsne_result[:, 2],
#     text=labels,
#     title="3D t-SNE of OpenAI Embeddings"
# )
# fig_tsne.show()

Example 2¶

It took me a lot of time to come up with one perfect example to demonstrate th

Embeddings I – Exploring Embeddings