embeddings_livnlearnversion

The purpose of this notebook is to study and evaluate the use of an embedding model in user segmentation

Task¶

User segmentation / profiling (without training)

Data¶

lets say if i have data for what locations people are interested in (maybe they’ve searched those or visited those)
i’ve formatted the data like:
“{user1}”: {
“{name of place} {city} {state} {country}”: {frequency of search/visit},
“{name of place} {city} {state} {country}”: {frequency of search/visit},
…
}

Plan¶

we’ve all spent time doing strenous feature selection and engineering using distribution tests, predictive powers, iterations etc etc to select key features, encode them with things like OHE and train models creating user vectors and clustering

but what if instead we could just get raw data encoded directly and user vectors created with context of each feature making user itself queryable- no predefined feature engg, vectorization or clustering – we use the power of pre trained embedding models to bring in the knowledge of context

Here’s how the process could look like:

get distinct strings (lets call them domain entries)
get embedding for each distinct domain entry – {name of place} {city} {state} {country} – using openai embedding model
weigh each embedding by frequency and average them together to create a user vector
encode query vector using same embedding model
list all users similar to a description (using a cosine similarity for instance)

Purpose of this notebook¶

To test the embedding model and get more insights on how they encode – what can we do with their embeddings, characterstics of embedding spaces – how does doing a weighted average look like in principle

concerns in real life behaviour could have tens of thousands of distinct placecodes for a user, combining them could lead to things like

dilution by too frequent behaviours, too many domains
dilution of weaker traits by combination
information loss
missclassification etc. etc. I discuss them below in the Test section

We use Open AI text embedding 3 small to inspect and play with the embeddings
To begin with, the import section below imports the necessary libraries and sets up the embedding function

Imports¶

In [50]:

from dotenv import load_dotenv
import os

import pandas as pd
import numpy as np

from sklearn.metrics.pairwise import cosine_similarity
from sklearn.cluster import KMeans

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

from matplotlib import pyplot as plt
import seaborn as sns
import plotly.express as px
import plotly.io as pio
pio.renderers.default = "notebook"

In [51]:

import openai
from openai import OpenAI

load_dotenv()
openai_api_key = os.getenv("OPENAI_API_KEY")

openai_client = OpenAI(
    api_key=os.getenv("OPENAI_API_KEY")
)
def get_embedding(text, deployment_name="text-embedding-3-small"):
    response = openai_client.embeddings.create(
        input=[text],
        model=deployment_name
    )
    return response.data[0].embedding

Example 1¶

We start with the standard example they quote everywhere – basically if the context makes sense and its the same vector space, then king – man + woman -> queen

In [52]:

texts = {
    "king": "king",
    "man": "man",
    "woman": "woman",
    "queen": "queen"
}

In [53]:

embeddings = {k: np.array(get_embedding(v)) for k, v in texts.items()}

In [54]:

king_vec = embeddings["king"]
man_vec = embeddings["man"]
woman_vec = embeddings["woman"]
queen_vec = embeddings["queen"]

In [55]:

analogy_vec = king_vec - man_vec + woman_vec

In [56]:

similarity = cosine_similarity([analogy_vec], [queen_vec])[0][0]
print(f"Cosine similarity with 'queen': {similarity:.4f}")

Cosine similarity with 'queen': 0.6162

In [57]:

print("Similarity of analogy vector to all texts:")
for word, vec in embeddings.items():
    sim = cosine_similarity([analogy_vec], [vec])[0][0]
    print(f"{word}: {sim:.4f}")

Similarity of analogy vector to all texts:
king: 0.7637
man: 0.1922
woman: 0.6023
queen: 0.6162

Even though this is quite similar to queen (& woman) – its still most similar to king

In [58]:

embeddings['analogy'] = analogy_vec
labels = list(embeddings.keys())
embedding_matrix = np.array(list(embeddings.values()))

you could choose either PCA or t-SNE to visualize both will allow to reduce this 1536 dimension to a 2-3 dim one for visualization ( its important to note they’re approximations and both of them work differently – inspite of what people keep saying :/ ) – in my experience tSNE better captures this higher dimensionality but you’re free to try both – this case is very simple so it doesn’t matter but you can add more examples and see

In [59]:

pca_3d = PCA(n_components=3)
pca_result = pca_3d.fit_transform(embedding_matrix)

fig_pca = px.scatter_3d(
    x=pca_result[:, 0], y=pca_result[:, 1], z=pca_result[:, 2],
    text=labels,
    title="3D PCA of OpenAI Embeddings"
)
fig_pca.show()

In [60]:

# tsne_3d = TSNE(n_components=3, perplexity=3, n_iter=1000, random_state=42)
# tsne_result = tsne_3d.fit_transform(embedding_matrix)

# fig_tsne = px.scatter_3d(
#     x=tsne_result[:, 0], y=tsne_result[:, 1], z=tsne_result[:, 2],
#     text=labels,
#     title="3D t-SNE of OpenAI Embeddings"
# )
# fig_tsne.show()

Example 2¶

It took me a lot of time to come up with one perfect example to demonstrate the potential challenges i wanted – ofcourse 2 hours on chatgpt backn forth helped

In [61]:

user_searches = {
    # Strong and coherent interest: SKI
    "user_ski_only": {
        "Ski Aspen Colorado United States": 6,
        "Ski Whistler British Columbia Canada": 5,
        "Ski St Anton Tyrol Austria": 5
    },

    # Strong interest diluted by frequent unrelated behaviour (groceries, coffee shops)
    "user_ski_with_daily_life": {
        "Ski Aspen Colorado United States": 6,
        "Ski Whistler British Columbia Canada": 5,
        "Ski St Anton Tyrol Austria": 5,
        "Starbucks New York New York United States": 30,
        "Trader Joe's San Francisco California United States": 25,
        "Whole Foods Austin Texas United States": 20
    },

    # Weak trait: Taylor Swift fan (only 1–2 relevant signals)
    "user_taylor_swift_fan": {
        "Taylor Swift Eras Tour Los Angeles California United States": 2,
        "Nail Salon Brooklyn New York United States": 3,
        "Pop Music Club Miami Florida United States": 2
    },

    # Taylor Swift fan diluted by unrelated high-frequency daily searches
    "user_taylor_swift_with_noise": {
        "Taylor Swift Eras Tour Los Angeles California United States": 2,
        "Nail Salon Brooklyn New York United States": 3,
        "Pop Music Club Miami Florida United States": 2,
        "Starbucks Chicago Illinois United States": 20,
        "Walmart Phoenix Arizona United States": 18,
        "Home Depot Houston Texas United States": 15,
        "CVS Pharmacy Boston Massachusetts United States": 12
    },

    # Multi-interest user: SKI + Tennis + Horror
    "user_ski_tennis_horror": {
        "Ski Aspen Colorado United States": 4,
        "City St Anton Tyrol Austria": 4,
        "Tennis Court Queens New York United States": 4,
        "City Wimbledon London United Kingdom": 4,
        "Horror Nights Universal Orlando Florida United States": 4,
        "Haunted House Salem Massachusetts United States": 4
    },

    # Multi-interest + unrelated noise (risk of cancellation / misclassification)
    "user_ski_tennis_horror_with_noise": {
        "Ski Aspen Colorado United States": 4,
        "City St Anton Tyrol Austria": 4,
        "Tennis Court Queens New York United States": 4,
        "City Wimbledon London United Kingdom": 4,
        "Horror Nights Universal Orlando Florida United States": 4,
        "Haunted House Salem Massachusetts United States": 4,
        "Grocery Store Dallas Texas United States": 30,
        "Gas Station Atlanta Georgia United States": 25,
        "Starbucks Los Angeles California United States": 20,
        "Walmart Denver Colorado United States": 15
    },

    # Pure noise: No clear theme
    "user_random_behaviour": {
        "Starbucks New York New York United States": 20,
        "McDonald's Chicago Illinois United States": 15,
        "Walmart Los Angeles California United States": 18,
        "Grocery Store Miami Florida United States": 12,
        "Gym Boston Massachusetts United States": 5,
        "Gas Station Seattle Washington United States": 8,
        "Pharmacy San Diego California United States": 10
    },

    "user_ski_horror_romcom": {
        "Ski Aspen Colorado United States": 4,
        "Ski Zermatt Valais Switzerland": 4,
        "Horror Nights Universal Orlando Florida United States": 4,
        "Haunted House Salem Massachusetts United States": 4,
        "Romantic Comedy Theatre New York New York United States": 4,
        "Romantic Movies Los Angeles California United States": 4
    },
    "user_luxury": {
        "Fairmont Chateau Whistler British Columbia Canada": 1,
        "Pebble Beach Golf Links California United States": 2,
        "Don Alfonso Toronto Canada": 2,
    },
    "user_party_parent": {
        "Montessori School New York New York United States": 6,
        "Children's Library New York New York United States": 6,
        "Night Club Las Vegas Nevada United States": 6,
        "Rooftop Bar New York New York United States": 6
    }


}

Create user vectors¶

domain_entry embeddings¶

In [62]:

all_domain_entrys = set(pc for user in user_searches.values() for pc in user)
# print(f"All unique domain_entrys: {all_domain_entrys}")
domain_entry_embeddings = {pc: np.array(get_embedding(pc)) for pc in all_domain_entrys}
# len(domain_entry_embeddings[""])

In [63]:

len(domain_entry_embeddings["Fairmont Chateau Whistler British Columbia Canada"])

Out[63]:

Weighted average¶

Why this works:

where we add up v embeddings for N places multiplying each with its respective frequency f and averaging into a single vector for a user u

Weighted averaging as you know and can see is linear (assumes linear relationships with all variables) – which is what makes it challenging

Live & Learn

Embeddings I – Exploring Embeddings