Rstats 101

How to Use Embeddings and Semantic Search in R

Fri, 10 Apr 2026 18:30:00 GMT

Introduction

Computers don’t really understand words. They understand numbers. Embeddings are a way to convert text into numbers so a computer can work with meaning, not just exact words.

Imagine this. You write the sentence “The cat is sleeping on the couch.” A model turns it into a list of numbers.

Every model has its own fixed vector size. A sentence fed into text-embedding-3-small always comes back as 1,536 numbers, and every input sentence produces exactly the same number of values. That fixed size is what lets you compare any two embeddings with simple math.

How long is that list? It depends on the model. Here are a few popular ones:

Model	Provider	Dimensions	Notes
`text-embedding-3-small`	OpenAI	1,536	Cheap default, high quality
`text-embedding-3-large`	OpenAI	3,072	Best OpenAI accuracy
`text-embedding-004`	Google Gemini	768	Free tier available
`voyage-3`	Voyage AI	1,024	Strong for retrieval
`embed-english-v3.0`	Cohere	1,024	Multilingual option available
`nomic-embed-text`	Ollama (local)	768	Free, runs offline
`mxbai-embed-large`	Ollama (local)	1,024	Higher quality local model
`all-MiniLM-L6-v2`	Sentence Transformers	384	Tiny, fast, surprisingly good

Now you write “A kitten is resting on the sofa.” Even though the words are different, the meaning is very similar, so the lists of numbers will also be very similar.

But if you compare it with “Stock markets fell sharply today,” the numbers will be very different, because the meaning is unrelated.

This simple idea is powerful. Instead of matching exact keywords, you can match meaning.

For example:

If you search for “comfortable place to sit”, you can find results like “sofa” or “armchair”
You can detect that “cheap flights” and “low-cost airfare” mean the same thing
You can group documents about “diabetes”, “insulin”, and “blood sugar” together automatically

In the past, doing this required complicated rules or custom models. With embeddings, you just compare how “close” two pieces of text are.

In this tutorial, you’ll learn how to:

Turn text into embeddings using R and the OpenAI API
Measure how similar two pieces of text are
Build a simple semantic search system for your own documents

What you can build with embeddings:

Search a knowledge base by meaning, not keywords
Find duplicate or near-duplicate questions
Cluster documents by topic
Recommend related articles
Power the retrieval step in RAG systems

How Embeddings Work

A text embedding is a vector of floating-point numbers (often 768, 1024, or 1536 dimensions). The model is trained so that semantically similar text lands near each other in this vector space.

Approach	Matches On	Handles Synonyms?
Keyword search	Exact words	No
Embedding search	Meaning	Yes

For example, “How do I reset my password?” and “I forgot my login credentials” share almost no words but produce very similar embeddings.

The Famous King–Queen Example

The classic intuition for embeddings comes from the original word2vec paper. When you embed individual words, related words cluster together in predictable ways:

king is close to queen
man is close to woman
Paris is close to France
dog is close to puppy

But the surprising part is that the directions between words also carry meaning. If you take the vector for king, subtract the vector for man, and add the vector for woman, you land very close to the vector for queen:

king − man + woman ≈ queen

The same trick works for capitals:

Paris − France + Italy ≈ Rome

The model never saw an explicit rule about gender or geography. It learned these patterns from raw text alone, and the geometry of the vector space ended up encoding them. This is what people mean when they say embeddings “capture meaning” — relationships between concepts become directions in the space.

Pair A	Pair B	Shared Direction
man → woman	king → queen	gender
France → Paris	Italy → Rome	country → capital
walk → walked	run → ran	present → past tense
big → bigger	small → smaller	comparative form

Modern sentence-level embeddings (like the ones from OpenAI) are far richer than word2vec, but they’re built on the same core idea: meaning becomes geometry, and geometry is something a computer can work with using simple math like cosine similarity.

Getting Started

You’ll need an API key for an embeddings provider. OpenAI is the most common choice; we’ll use it here.

library(httr2)
library(tidyverse)

Sys.setenv(OPENAI_API_KEY = "your-key-here")

See How to Use the OpenAI API in R for setup details.

Get a Single Embedding

The OpenAI embeddings endpoint takes text and returns a numeric vector. Here’s a small helper:

embed_text <- function(text, model = "text-embedding-3-small") {
  request("https://api.openai.com/v1/embeddings") |>
    req_auth_bearer_token(Sys.getenv("OPENAI_API_KEY")) |>
    req_body_json(list(input = text, model = model)) |>
    req_perform() |>
    resp_body_json()
}

This wraps a single POST request. The text-embedding-3-small model is cheap and fast — a good default.

Call it on one string

result <- embed_text("R is a language for statistical computing")

vec <- unlist(result$data[[1]]$embedding)
length(vec)
# [1] 1536

The returned vector has 1536 numbers. Let’s peek at the first ten to see what they actually look like:

head(vec, 10)
# [1] -0.01823  0.04517 -0.00192  0.03104 -0.02276
# [6]  0.01438  0.00067 -0.02651  0.05209 -0.01102

Each number is a small floating-point value, typically between roughly -0.1 and 0.1. Individually these numbers mean nothing to a human — you can’t point at the third coordinate and say “that’s the ‘statistics’ dimension.” The meaning lives in the pattern of all 1536 values taken together, and you surface that meaning by comparing vectors to each other.

A quick summary of the distribution confirms this:

summary(vec)
#     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
# -0.10382  -0.01891   0.00024  -0.00006   0.01923   0.11047

The values are centered near zero and spread symmetrically — a typical normalized embedding. You don’t read them directly; you compare them to other vectors.

Cosine Similarity: Comparing Two Embeddings

The standard way to compare embeddings is cosine similarity — the cosine of the angle between two vectors. Values range from -1 (opposite) to 1 (identical direction).

cosine_sim <- function(a, b) {
  sum(a * b) / (sqrt(sum(a^2)) * sqrt(sum(b^2)))
}

Cosine similarity ignores vector length and only cares about direction, which is what you want when comparing meaning.

Compare two sentences

a <- unlist(embed_text("How do I reset my password?")$data[[1]]$embedding)
b <- unlist(embed_text("I forgot my login")$data[[1]]$embedding)
c <- unlist(embed_text("What's the weather today?")$data[[1]]$embedding)

cosine_sim(a, b)  # ~0.78 (similar meaning)
cosine_sim(a, c)  # ~0.12 (unrelated)

The first pair scores high despite sharing no words. That’s the embedding magic.

Embed Many Documents at Once

The OpenAI API accepts a list of inputs in a single call. Batching is much cheaper than one call per document.

embed_many <- function(texts, model = "text-embedding-3-small") {
  result <- request("https://api.openai.com/v1/embeddings") |>
    req_auth_bearer_token(Sys.getenv("OPENAI_API_KEY")) |>
    req_body_json(list(input = texts, model = model)) |>
    req_perform() |>
    resp_body_json()

  map(result$data, \(d) unlist(d$embedding))
}

Build an embedding store

docs <- c(
  "R is a language for statistical computing",
  "Python is popular for data science",
  "ggplot2 makes beautiful charts",
  "Pizza is a popular Italian dish",
  "Use dplyr to filter and group data"
)

embeddings <- embed_many(docs)
length(embeddings)  # 5

You now have one vector per document. Store them in a tibble alongside the original text.

store <- tibble(
  text = docs,
  embedding = embeddings
)

store
# # A tibble: 5 × 2
#   text                                      embedding
#                                        
# 1 R is a language for statistical computing 
# 2 Python is popular for data science        
# 3 ggplot2 makes beautiful charts            
# 4 Pizza is a popular Italian dish           
# 5 Use dplyr to filter and group data

Notice the embedding column is a list-column — each row holds an entire 1,536-element numeric vector. This is one of the most useful patterns in the tidyverse: keep complex objects inside a tibble so you can treat them as ordinary rows. If list-columns are new to you, see How to Use nest() in R for the broader pattern.

You can peek at the first few values of any row with pluck():

store$embedding[[1]] |> head(5)
# [1] -0.01823  0.04517 -0.00192  0.03104 -0.02276

Keeping the text and embedding side-by-side makes search results easy to display.

Visualizing Semantic Similarity

Numbers are abstract — plots make the geometry of embeddings concrete. Two visualizations are especially useful: a similarity heatmap showing how every document relates to every other, and a 2D projection placing the documents on a plane so you can literally see the clusters.

Build a Similarity Matrix

First, compute cosine similarity between every pair of documents in the store. The result is a square matrix where row i, column j is the similarity between document i and document j.

similarity_matrix <- function(embeddings) {
  n <- length(embeddings)
  m <- matrix(0, n, n)
  for (i in seq_len(n)) {
    for (j in seq_len(n)) {
      m[i, j] <- cosine_sim(embeddings[[i]], embeddings[[j]])
    }
  }
  m
}

sim <- similarity_matrix(embeddings)
round(sim, 2)
#      [,1] [,2] [,3] [,4] [,5]
# [1,] 1.00 0.55 0.45 0.10 0.50
# [2,] 0.55 1.00 0.30 0.08 0.40
# [3,] 0.45 0.30 1.00 0.05 0.45
# [4,] 0.10 0.08 0.05 1.00 0.08
# [5,] 0.50 0.40 0.45 0.08 1.00

The diagonal is always 1 (every document is identical to itself). Off-diagonal cells reveal which documents the model thinks are related. Row 4 (the pizza sentence) stands out immediately — every off-diagonal value is below 0.11, confirming it’s unrelated to the other four. You can already see the story in the raw numbers; the heatmap just makes it visual.

For easier reading, label the rows and columns with short names and turn the matrix into a tibble:

short <- c("R", "Python", "ggplot2", "Pizza", "dplyr")
rownames(sim) <- short
colnames(sim) <- short

as_tibble(sim, rownames = "doc") |>
  mutate(across(where(is.numeric), \(x) round(x, 2)))
# # A tibble: 5 × 6
#   doc        R Python ggplot2 Pizza dplyr
#            
# 1 R       1     0.55    0.45  0.1    0.5
# 2 Python  0.55  1       0.3   0.08   0.4
# 3 ggplot2 0.45  0.3     1     0.05   0.45
# 4 Pizza   0.1   0.08    0.05  1      0.08
# 5 dplyr   0.5   0.4     0.45  0.08   1

Plot a Heatmap with ggplot

To plot it with ggplot, convert the matrix to a long-format data frame with one row per pair. This uses pivot_longer() to reshape the matrix:

sim_long <- as_tibble(sim) |>
  set_names(docs) |>
  mutate(doc_a = docs) |>
  pivot_longer(-doc_a, names_to = "doc_b", values_to = "similarity")

Now draw the heatmap with geom_tile(), which is the standard ggplot2 layer for matrix-style visualizations:

ggplot(sim_long, aes(doc_a, doc_b, fill = similarity)) +
  geom_tile(color = "white") +
  geom_text(aes(label = round(similarity, 2)), size = 3) +
  scale_fill_gradient2(
    low = "white", mid = "lightblue", high = "steelblue",
    midpoint = 0.5, limits = c(0, 1)
  ) +
  labs(
    title = "Semantic Similarity Between Documents",
    x = NULL, y = NULL, fill = "Cosine"
  ) +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 35, hjust = 1))

Heatmap of cosine similarity between text embeddings in R using ggplot2, showing data-science sentences clustering together and the pizza sentence as an outlier

What you’ll see: The two R-related sentences (“R is a language…” and “Use dplyr…”) form a bright cell, “ggplot2 makes beautiful charts” lights up against both R sentences, “Python is popular for data science” sits closer to the R sentences than to the pizza sentence, and “Pizza is a popular Italian dish” stays a pale outlier connected to nothing. The heatmap visually proves the model has grouped the data-science sentences together — without ever being told what data science is.

Plot a 2D Projection

A heatmap shows pairwise relationships, but a scatter plot shows the underlying geometry. Embeddings live in hundreds of dimensions, so we use PCA to squash them into two dimensions for plotting.

emb_matrix <- do.call(rbind, embeddings)

pca <- prcomp(emb_matrix, center = TRUE, scale. = FALSE)

points <- tibble(
  text = docs,
  x = pca$x[, 1],
  y = pca$x[, 2]
)

prcomp() finds the two directions in the embedding space that explain the most variance. The first two principal components are usually enough to reveal clusters.

Plot the projection

ggplot(points, aes(x, y, label = text)) +
  geom_point(size = 4, color = "steelblue") +
  geom_text(nudge_y = 0.05, size = 3.5) +
  labs(
    title = "Documents Projected to 2D with PCA",
    x = "PC1", y = "PC2"
  ) +
  theme_minimal() +
  expand_limits(x = c(-1, 1), y = c(-0.5, 0.5))

PCA 2D projection of text embeddings in R with ggplot2, showing four data-science documents clustered together and the pizza document isolated on the right

What you’ll see: the four data-science sentences cluster together on one side of the plot, while the pizza sentence sits alone on the other side. The model has effectively drawn a line between “things about data” and “things about food” — based purely on the geometry of the embeddings.

Visualize Query Distance

You can also use the same plot to show how close a search query is to each document. Embed the query, add a score column with mutate() and map_dbl(), then color the points by that score:

query <- "how do I plot data in R?"
q_vec <- unlist(embed_text(query)$data[[1]]$embedding)

points |>
  mutate(score = map_dbl(embeddings, \(e) cosine_sim(q_vec, e))) |>
  ggplot(aes(x, y, label = text, color = score, size = score)) +
  geom_point() +
  geom_text(nudge_y = 0.05, size = 3.5, color = "black") +
  scale_color_gradient(low = "grey80", high = "darkred") +
  labs(title = paste0("Similarity to query: '", query, "'")) +
  theme_minimal()

Semantic search query distance plot in R with ggplot2, showing documents sized and colored by cosine similarity to a user query about plotting data

The points closest to the query (in meaning) will be the largest and darkest. This is what semantic search actually looks like under the hood.

Anti-Correlation: When Sentences Disagree

So far we’ve focused on documents that should land close together. Equally important is seeing what happens when sentences are unrelated — the low-similarity end of the scale. This is what makes semantic search trustworthy: irrelevant results need to be visibly far away, not just slightly less close.

A Mixed Set of Sentences

Build a small set that deliberately mixes related topics and totally unrelated ones:

mixed <- c(
  "I love this product, it works perfectly",
  "This is the best thing I have ever bought",
  "I hate this product, it broke immediately",
  "Worst purchase I have ever made",
  "Quantum chromodynamics describes the strong force",
  "The Higgs boson was discovered in 2012"
)

mixed_emb <- embed_many(mixed)
mixed_sim <- similarity_matrix(mixed_emb)

Three groups: positive reviews (1–2), negative reviews (3–4), and physics statements (5–6). We expect physics to be far from everything else, and reviews to cluster — but with an interesting twist around antonyms.

Plot the Mixed Heatmap

labels <- c("love it", "best ever", "hate it", "worst ever", "QCD", "Higgs")

mixed_long <- as_tibble(mixed_sim) |>
  set_names(labels) |>
  mutate(doc_a = labels) |>
  pivot_longer(-doc_a, names_to = "doc_b", values_to = "similarity")

ggplot(mixed_long, aes(doc_a, doc_b, fill = similarity)) +
  geom_tile(color = "white") +
  geom_text(aes(label = round(similarity, 2)), size = 3) +
  scale_fill_gradient2(
    low = "firebrick", mid = "white", high = "steelblue",
    midpoint = 0.5, limits = c(0, 1)
  ) +
  labs(
    title = "Correlation and Anti-Correlation",
    subtitle = "Reviews cluster together; physics is far from both",
    x = NULL, y = NULL, fill = "Cosine"
  ) +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 35, hjust = 1))

Heatmap showing correlation and anti-correlation between text embeddings in R, with product reviews forming one block and physics statements forming a separate block with pale across-block cells

What you’ll see: Two bright blocks form along the diagonal — one for the review pair, one for the physics pair. Between them, the cells turn pale: reviews vs physics scores around 0.05–0.15, the lowest values in the matrix. That low score is what “anti-correlation” looks like in practice for embeddings — not negative numbers, but a clear visual gap.

The Surprising Antonym Result

Run this query and watch what happens:

cosine_sim(mixed_emb[[1]], mixed_emb[[3]])
# ~0.72  ("love it" vs "hate it")

cosine_sim(mixed_emb[[1]], mixed_emb[[5]])
# ~0.08  ("love it" vs "QCD")

“I love this product” and “I hate this product” score higher than “I love this product” and a physics sentence. That’s surprising at first — but it makes sense: both reviews share grammar, vocabulary, the word “product”, and the entire context of a customer opinion. The embedding captures topic much more strongly than sentiment.

This is a critical lesson:

Embedding similarity measures topical relatedness, not agreement. Two sentences that contradict each other will often score high if they’re about the same thing.

If you need to detect agreement vs disagreement (e.g., for fact-checking or stance detection), embeddings alone won’t do it — you need an LLM with a classification prompt. See How to Classify Text with LLMs in R for that approach.

Visualize the Three Clusters

The PCA scatter plot makes the topic vs sentiment distinction obvious:

mixed_pca <- prcomp(do.call(rbind, mixed_emb), center = TRUE)

tibble(
  text = labels,
  group = c("review", "review", "review", "review", "physics", "physics"),
  x = mixed_pca$x[, 1],
  y = mixed_pca$x[, 2]
) |>
  ggplot(aes(x, y, color = group, label = text)) +
  geom_point(size = 5) +
  geom_text(nudge_y = 0.05, color = "black", size = 3.5) +
  scale_color_manual(values = c(review = "steelblue", physics = "firebrick")) +
  labs(
    title = "Topic Beats Sentiment in Embedding Space",
    subtitle = "Positive and negative reviews cluster together; physics sits far away",
    x = "PC1", y = "PC2"
  ) +
  theme_minimal()

PCA scatter plot of text embeddings in R with ggplot2 showing topic dominates sentiment, with positive and negative product reviews clustered together and physics statements far away

What you’ll see: four reviews forming one tight blue cluster (positive and negative mixed together) and two physics sentences forming a separate red cluster on the other side of the plot. The reviews aren’t separated by sentiment — they’re unified by topic. That single picture tells you exactly what embeddings are good at and what they’re not.

Build a Semantic Search Function

Searching means: embed the query, compare it to every stored embedding, return the top matches.

semantic_search <- function(query, store, top_n = 3) {
  q_vec <- unlist(embed_text(query)$data[[1]]$embedding)

  store |>
    mutate(score = map_dbl(embedding, \(e) cosine_sim(q_vec, e))) |>
    arrange(desc(score)) |>
    head(top_n)
}

The pipeline uses mutate() to add a score column, arrange() to sort by descending similarity, and head() to keep the top matches. If these dplyr verbs are new to you, start with How to Use mutate() in R.

Run a query

semantic_search("how to make plots in R", store, top_n = 2)

text	score
ggplot2 makes beautiful charts	0.71
R is a language for statistical computing	0.55

Notice that the top match doesn’t contain the word “plots” — it matches on meaning.

Practical Example: Search FAQ Entries

A common real-world use is searching a help center or FAQ. Build the store once, query it many times.

faqs <- c(
  "To reset your password, click 'Forgot password' on the login page",
  "You can cancel your subscription in Account Settings",
  "We accept Visa, Mastercard, and PayPal",
  "Refunds are processed within 5-7 business days",
  "Two-factor authentication can be enabled in Security Settings"
)

faq_store <- tibble(
  text = faqs,
  embedding = embed_many(faqs)
)

Query the FAQ store

semantic_search("I can't log in", faq_store, top_n = 1)
# "To reset your password, click 'Forgot password'..."

semantic_search("how do I get my money back", faq_store, top_n = 1)
# "Refunds are processed within 5-7 business days"

A keyword search would miss both of these. The embedding search handles paraphrases naturally.

From Search to RAG: Answer Questions Over Your Docs

Semantic search returns relevant passages, but it doesn’t answer questions. That final step — feeding the retrieved passages into an LLM along with the user’s question — is called retrieval-augmented generation, or RAG. It’s the most common way to build an AI assistant grounded in your own documents.

The recipe is three steps:

Retrieve — use semantic search to find the top-k passages related to the question
Augment — build a prompt that includes those passages as context
Generate — ask an LLM to answer the question using only that context

You already have step 1. Steps 2 and 3 are just string assembly and one chat call.

The RAG Function

Here’s a complete RAG function built on the semantic_search() and faq_store from earlier:

library(ellmer)

answer_with_rag <- function(question, store, top_k = 3) {
  hits <- semantic_search(question, store, top_n = top_k)

  context <- paste(
    "Passage", seq_len(nrow(hits)), ":", hits$text,
    collapse = "\n\n"
  )

  prompt <- paste(
    "Answer the question using ONLY the passages below.",
    "If the answer isn't there, say 'I don't know.'",
    "\n\nContext:\n", context,
    "\n\nQuestion:", question
  )

  chat <- chat_claude()
  chat$chat(prompt)
}

The function retrieves the top_k best passages, packages them as labeled context, and instructs the LLM to answer only from that context. The “say I don’t know” clause is critical — without it, the model will fall back to its own training data when the retrieval fails.

Run a RAG Query

answer_with_rag("I can't log in to my account, what should I do?", faq_store)
# "To reset your password, click 'Forgot password' on the login page."

answer_with_rag("what's the capital of France?", faq_store)
# "I don't know."

Notice the second response — the question is unrelated to any FAQ entry, so the model refuses rather than hallucinating. That refusal is the whole point of RAG: ground the answer in your data and fail loudly when the data doesn’t cover the question.

Why Retrieval Beats a Huge Prompt

You might ask: why not just paste all the FAQs into every prompt and skip retrieval entirely? Two reasons:

Approach	Cost per query	Scales to
Dump everything	Grows with doc count	~50–100 short docs
RAG with retrieval	Fixed (top-k only)	Millions of docs

Retrieval keeps the prompt small, which makes responses faster, cheaper, and more accurate. LLMs also get distracted by irrelevant context, so handing them only the top matches usually improves answer quality.

Going Further with RAG

A few patterns worth knowing once the basic version works:

Chunk long documents before embedding. Split a 10-page PDF into ~300-word chunks so retrieval can find the specific paragraph that answers a question.
Include metadata (source URL, section title) in the context so the LLM can cite where the answer came from.
Re-rank retrieved results with a cross-encoder or a second LLM pass for higher precision at the cost of latency.
Use a vector database (pgvector, Qdrant, LanceDB) once you have more than a few thousand documents.

This is enough to build a real “chat with your docs” app in R. For a deeper dive into chat APIs, see How to Use ellmer in R and How to Use the Claude API in R.

Caching Embeddings

Embeddings are deterministic for a given text and model, so you should cache them. Rebuilding embeddings on every script run wastes money and time.

cache_path <- "embeddings_cache.rds"

if (file.exists(cache_path)) {
  store <- read_rds(cache_path)
} else {
  store <- tibble(
    text = docs,
    embedding = embed_many(docs)
  )
  write_rds(store, cache_path)
}

For larger projects, store embeddings in a database column or a dedicated vector store.

Choosing a Model

OpenAI offers several embedding models with different sizes and costs.

Model	Dimensions	Best For
`text-embedding-3-small`	1536	Default — cheap and good
`text-embedding-3-large`	3072	Highest quality
`text-embedding-ada-002`	1536	Legacy, avoid for new work

Start with text-embedding-3-small. Move to large only if you measure a meaningful quality gain — it costs about 6x more and produces vectors twice as large.

Use Local Embeddings with Ollama

If you don’t want to send data to a third-party API, run embeddings locally with Ollama.

embed_local <- function(text, model = "nomic-embed-text") {
  request("http://localhost:11434/api/embeddings") |>
    req_body_json(list(model = model, prompt = text)) |>
    req_perform() |>
    resp_body_json()
}

The interface is similar — only the URL and request shape change. See How to Run Local LLMs in R for Ollama setup.

Note: Local embedding models are typically smaller (768 dimensions) and may be less accurate than cloud models, but they’re free and private.

Performance Tips

Batch your requests. One call with 100 documents is dramatically faster and cheaper than 100 separate calls.

Cache aggressively. The same input always produces the same vector, so persist them to disk or a database.

Normalize once if you do many comparisons. If you precompute the L2 norm of each vector, cosine similarity becomes a dot product, which is faster.

normalize <- function(v) v / sqrt(sum(v^2))

store_norm <- store |>
  mutate(embedding = map(embedding, normalize))

# Now cosine_sim simplifies to sum(a * b)

For thousands of documents this is fine in pure R. For millions, you’ll want a vector database (pgvector, Qdrant, Pinecone).

Common Mistakes

1. Comparing vectors from different models

Embeddings are only meaningful within the same model. A vector from text-embedding-3-small cannot be compared to one from text-embedding-3-large — they live in different spaces.

2. Forgetting to cache

Re-embedding the same documents on every run is the most common waste of money in embedding projects. Cache by text content or hash.

3. Treating cosine score as a probability

A score of 0.7 doesn’t mean “70% confident.” It just means “more similar than 0.5 and less similar than 0.9.” Use it for ranking, not absolute thresholds — and if you do threshold, calibrate on your own data.

Summary

Step	Function
Embed a single text	`embed_text(text)`
Embed many texts	`embed_many(texts)`
Compare two embeddings	`cosine_sim(a, b)`
Search a store	`semantic_search(query, store)`

Key points:

Embeddings turn text into vectors that capture meaning
Cosine similarity is the standard comparison metric
Batch your requests and cache the results
Use text-embedding-3-small as a default starting model
Embeddings are the foundation for semantic search and RAG

Frequently Asked Questions

How much do embeddings cost in R?

OpenAI’s text-embedding-3-small costs about $0.02 per 1 million tokens (roughly 750,000 words). For most projects this is negligible — embedding an entire 10,000-entry FAQ typically costs less than a cent. The text-embedding-3-large model is ~6x more expensive but produces higher-quality vectors. For free alternatives, use Ollama with nomic-embed-text.

Do I need a vector database to use embeddings in R?

No — not until you have tens of thousands of documents. For small and medium projects, a tibble with an embedding list-column is completely fine and cosine similarity in pure R is fast enough. Reach for a vector database (pgvector, Qdrant, LanceDB, DuckDB’s vss extension) only when in-memory comparison becomes a bottleneck, typically past a few hundred thousand documents.

Can I use open-source embedding models in R?

Yes. The easiest path is Ollama, which serves models like nomic-embed-text or mxbai-embed-large through a local HTTP API. You can also use Hugging Face models via reticulate and the sentence-transformers Python library. Local models are free, private, and work offline — at the cost of some accuracy and setup time.

How do I update embeddings when my documents change?

Embeddings are deterministic for a given text + model, so you only need to re-embed documents whose content has changed. The simplest approach is to hash each document and store the hash alongside its embedding; if the hash changes, re-embed that row. This lets you sync a large corpus with a tiny incremental cost.

How many dimensions should my embeddings have?

Default to whatever the model produces — text-embedding-3-small gives 1536 dimensions and that’s fine for almost all use cases. OpenAI lets you truncate with a dimensions parameter (e.g., 512) to save storage, and the newer models are trained so that truncation preserves most of the quality. Only tune this if storage or memory is genuinely a bottleneck.

Are embeddings enough for sentiment analysis?

No — and this is a common pitfall. Embedding similarity captures topic, not agreement. Two sentences that say opposite things about the same subject often score high. For sentiment or stance detection, use an LLM with a classification prompt instead. See How to Analyze Sentiment with LLMs in R and How to Classify Text with LLMs in R.

What’s the difference between embeddings and fine-tuning?

Embeddings work with a frozen pre-trained model — you don’t change the model, you just use its output. Fine-tuning actually updates the model weights on your own data. For the vast majority of “search my docs” and “answer questions from my data” problems, embeddings + RAG are cheaper, faster, and easier to maintain than fine-tuning, while giving you the ability to update your knowledge base without retraining anything.

Can I build a chatbot with embeddings in R?

Yes — that’s exactly what the RAG section above shows. Combine semantic search with a chat call to Claude or OpenAI, wrap it in a Shiny app, and you have a “chat with your docs” application running entirely in R.

Sources

How to Analyze Sentiment with LLMs in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

Sentiment analysis determines the emotional tone of text - is it positive, negative, or neutral? LLMs make this easy without training custom models.

Common use cases:

Analyze customer reviews
Monitor social media mentions
Process survey feedback
Track brand sentiment over time

Why Use LLMs for Sentiment Analysis?

Approach	Setup	Nuance	Context Understanding
Rule-based	Word lists	Limited	Poor
Traditional ML	Training data	Moderate	Moderate
LLMs	Just a prompt	Excellent	Excellent

LLMs understand sarcasm, context, and nuance that simpler methods miss.

Getting Started

Load the packages:

library(ellmer)
library(tidyverse)

This tutorial uses ellmer which works with Claude, OpenAI, Gemini, or local models.

Basic Sentiment Analysis

Simple three-way classification

Start with positive, negative, or neutral:

chat <- chat_claude()

review <- "The food was delicious but the service was incredibly slow."

response <- chat$chat(paste(
  "Analyze the sentiment of this text.",
  "Reply with only: positive, negative, or neutral.",
  "\n\nText:", review
))

response
# "neutral"

The review has both positive (delicious food) and negative (slow service) elements, so the overall sentiment is neutral.

Why LLMs handle nuance better

Traditional sentiment tools often miss context:

# Sarcasm - sounds positive, means negative
"Oh great, another meeting that could have been an email."
# Rule-based: "great" = positive ❌
# LLM: negative ✓

# Negation - negative word, positive meaning
"Not bad at all, actually quite impressed."
# Rule-based: "bad" = negative ❌
# LLM: positive ✓

# Mixed - needs context weighing
"The product broke after a week, but customer service was amazing."
# LLM understands both aspects

Structured Sentiment Output

Using extract_data for reliable results

For consistent, parseable output, use structured extraction:

sentiment_type <- type_enum(
  values = c("positive", "negative", "neutral"),
  description = "Overall sentiment of the text"
)

The type_enum() guarantees the response is one of the three allowed values.

Extract sentiment

chat <- chat_claude()

review <- "Absolutely love this product! Best purchase I've made all year."

sentiment <- chat$extract_data(review, type = sentiment_type)

sentiment
# "positive"

Sentiment with Confidence Scores

When you need more than just a label

Get confidence scores to identify borderline cases:

sentiment_detailed <- type_object(
  sentiment = type_enum(
    values = c("positive", "negative", "neutral"),
    description = "Overall sentiment"
  ),
  confidence = type_number("Confidence score from 0 to 1"),
  explanation = type_string("Brief explanation")
)

Extract detailed sentiment

chat <- chat_claude()

review <- "It's okay I guess. Does what it says but nothing special."

result <- chat$extract_data(review, type = sentiment_detailed)

result$sentiment
# "neutral"

result$confidence
# 0.85

result$explanation
# "Lukewarm endorsement with no strong positive or negative indicators"

Low confidence (below 0.7) suggests the text is ambiguous and may need human review.

Fine-grained Sentiment

Five-point scale

For more granularity, use a five-point scale:

sentiment_5point <- type_enum(
  values = c("very_negative", "negative", "neutral", "positive", "very_positive"),
  description = "Sentiment on a 5-point scale"
)

Classify on the scale

chat <- chat_claude()

reviews <- c(
  "Terrible product. Complete waste of money. Avoid!",
  "Not great. Had some issues but usable.",
  "It's fine. Does the job.",
  "Really happy with this purchase!",
  "AMAZING! Exceeded all my expectations!!!"
)

classify_5point <- function(text) {
  chat <- chat_claude()
  Sys.sleep(0.5)
  chat$extract_data(text, type = sentiment_5point)
}

sentiments <- map_chr(reviews, classify_5point)

tibble(review = reviews, sentiment = sentiments)

review	sentiment
Terrible product…	very_negative
Not great…	negative
It’s fine…	neutral
Really happy…	positive
AMAZING!…	very_positive

Numeric scores

Convert to numbers for analysis:

sentiment_to_score <- function(sentiment) {
  scores <- c(
    very_negative = 1,
    negative = 2,
    neutral = 3,
    positive = 4,
    very_positive = 5
  )
  scores[sentiment]
}

df <- tibble(
  review = reviews,
  sentiment = sentiments,
  score = map_dbl(sentiments, sentiment_to_score)
)

mean(df$score)
# 3.0 (average sentiment)

Aspect-Based Sentiment

Analyze sentiment by aspect

Products often have multiple aspects with different sentiments:

aspect_sentiment <- type_object(
  overall = type_enum(
    values = c("positive", "negative", "neutral"),
    description = "Overall sentiment"
  ),
  aspects = type_array(
    items = type_object(
      aspect = type_string("The aspect being discussed"),
      sentiment = type_enum(
        values = c("positive", "negative", "neutral"),
        description = "Sentiment for this aspect"
      )
    ),
    description = "Individual aspects mentioned"
  )
)

Extract aspect sentiments

chat <- chat_claude()

review <- "The laptop is incredibly fast and the screen is beautiful.
However, the battery life is disappointing and the keyboard feels cheap."

result <- chat$extract_data(review, type = aspect_sentiment)

result$overall
# "neutral"

result$aspects
# [[1]] aspect: "speed", sentiment: "positive"
# [[2]] aspect: "screen", sentiment: "positive"
# [[3]] aspect: "battery", sentiment: "negative"
# [[4]] aspect: "keyboard", sentiment: "negative"

Convert to data frame

aspects_df <- tibble(
  aspect = map_chr(result$aspects, "aspect"),
  sentiment = map_chr(result$aspects, "sentiment")
)

aspects_df

aspect	sentiment
speed	positive
screen	positive
battery	negative
keyboard	negative

Batch Sentiment Analysis

Process multiple reviews

Create a reusable function:

analyze_sentiment <- function(text) {
  chat <- chat_claude()

  sentiment_type <- type_enum(
    values = c("positive", "negative", "neutral"),
    description = "Sentiment"
  )

  Sys.sleep(0.5)  # Rate limiting
  chat$extract_data(text, type = sentiment_type)
}

Apply to a data frame

reviews_df <- tibble(
  id = 1:5,
  review = c(
    "Love it! Works perfectly.",
    "Broken on arrival. Very disappointed.",
    "Does what it says. Nothing more.",
    "Best purchase ever! Highly recommend!",
    "Meh. Expected better for the price."
  )
)

reviews_df <- reviews_df |>
  mutate(sentiment = map_chr(review, analyze_sentiment))

reviews_df

id	review	sentiment
1	Love it!…	positive
2	Broken on arrival…	negative
3	Does what it says…	neutral
4	Best purchase…	positive
5	Meh. Expected…	negative

Calculate summary statistics

reviews_df |>
  count(sentiment) |>
  mutate(pct = n / sum(n) * 100)

sentiment	n	pct
positive	2	40%
negative	2	40%
neutral	1	20%

Domain-Specific Sentiment

Customize for your domain

Add context for better accuracy in specific domains:

chat <- chat_claude(
  system_prompt = "You analyze sentiment in restaurant reviews.

  Consider these aspects:
  - Food quality and taste
  - Service speed and friendliness
  - Ambiance and cleanliness
  - Value for money

  'Rich and decadent' is positive for desserts.
  'Simple and quick' is positive for lunch spots.
  'Crowded' can be positive (popular) or negative (uncomfortable)."
)

review <- "The place was packed on a Saturday night.
Food took 45 minutes but was worth the wait."

sentiment_type <- type_enum(
  values = c("positive", "negative", "neutral"),
  description = "Sentiment"
)

chat$extract_data(review, type = sentiment_type)
# "positive" (busy = popular, long wait but worth it)

Emotion Detection

Beyond positive/negative

Detect specific emotions:

emotion_type <- type_object(
  primary_emotion = type_enum(
    values = c("joy", "sadness", "anger", "fear", "surprise", "disgust", "neutral"),
    description = "Primary emotion expressed"
  ),
  intensity = type_enum(
    values = c("low", "medium", "high"),
    description = "Intensity of the emotion"
  )
)

Detect emotions

chat <- chat_claude()

texts <- c(
  "I can't believe they did this to me!",
  "Just got the promotion! So happy!",
  "The news was shocking, didn't see it coming."
)

detect_emotion <- function(text) {
  chat <- chat_claude()
  Sys.sleep(0.5)
  chat$extract_data(text, type = emotion_type)
}

emotions <- map(texts, detect_emotion)

tibble(
  text = texts,
  emotion = map_chr(emotions, "primary_emotion"),
  intensity = map_chr(emotions, "intensity")
)

text	emotion	intensity
I can’t believe…	anger	high
Just got the promotion…	joy	high
The news was shocking…	surprise	medium

Error Handling

Handle failures gracefully

safe_sentiment <- function(text) {
  tryCatch({
    chat <- chat_claude()
    sentiment_type <- type_enum(
      values = c("positive", "negative", "neutral"),
      description = "Sentiment"
    )
    Sys.sleep(0.5)
    chat$extract_data(text, type = sentiment_type)
  }, error = function(e) {
    warning("Sentiment analysis failed: ", e$message)
    NA_character_
  })
}

# Use with map
sentiments <- map_chr(reviews, safe_sentiment)

# Check for failures
sum(is.na(sentiments))

Local Sentiment Analysis

Use Ollama for free, private analysis

For sensitive data or high volume:

chat <- chat_ollama(model = "llama3.2")

sentiment_type <- type_enum(
  values = c("positive", "negative", "neutral"),
  description = "Sentiment"
)

chat$extract_data("Great product, highly recommend!", type = sentiment_type)
# "positive"

See Local LLMs with Ollama for setup instructions.

Comparison: LLM vs Traditional

When to use each approach

Scenario	Best Approach
Quick analysis, small dataset	LLM
Production, millions of texts	Traditional ML (faster, cheaper)
Need to understand sarcasm	LLM
Simple positive/negative	Either works
Multiple languages	LLM (built-in multilingual)
Sensitive data	Local LLM (Ollama)

Hybrid approach

Use LLMs to label training data, then train a faster model:

# Step 1: Label 1000 samples with LLM
labeled_data <- df |>
  slice_sample(n = 1000) |>
  mutate(sentiment = map_chr(text, analyze_sentiment))

# Step 2: Train a fast classifier on labeled data
# (using tidymodels or similar)

# Step 3: Use fast model for production

Summary

Task	Code
Basic sentiment	`chat$extract_data(text, type_enum(c("positive", "negative", "neutral")))`
With confidence	`type_object(sentiment = ..., confidence = type_number())`
5-point scale	`type_enum(c("very_negative", ..., "very_positive"))`
Aspect-based	`type_object(overall = ..., aspects = type_array(...))`
Emotion detection	`type_enum(c("joy", "anger", "sadness", ...))`

Key points:

LLMs understand sarcasm, context, and nuance
Use type_enum() for consistent output
Add domain context in system prompts for accuracy
Use Sys.sleep() in batch processing for rate limits
Consider local models for sensitive or high-volume data

Sources

How to Classify Text with LLMs in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

Text classification assigns categories to text. Traditional approaches require labeled training data and ML models. With LLMs, you can classify text with just a prompt - no training needed.

Common use cases:

Categorize support tickets (billing, technical, account)
Tag documents by topic (finance, legal, HR)
Classify emails by intent (inquiry, complaint, feedback)
Label survey responses by theme

Why Use LLMs for Classification?

Approach	Training Data	Setup Time	Flexibility
Traditional ML	Thousands of examples	Days/weeks	Fixed categories
LLMs	Zero examples	Minutes	Change categories anytime

LLMs excel when:

You don’t have labeled training data
Categories may change over time
You need quick prototyping

Getting Started

Load the ellmer package for LLM access:

library(ellmer)
library(tidyverse)

This tutorial works with any provider. See ellmer basics for setup.

Basic Classification

Single category classification

Start with a simple prompt that asks the LLM to classify text:

chat <- chat_claude()

text <- "I can't log into my account and I've tried resetting my password three times."

response <- chat$chat(paste(

  "Classify this support ticket into one category:",
  "billing, technical, account, or other.",
  "Reply with just the category name.",
  "\n\nTicket:", text
))

response
# "account"

The prompt has three parts:

Task: “Classify this support ticket”
Categories: “billing, technical, account, or other”
Format: “Reply with just the category name”

Why “just the category name”?

Without format instructions, LLMs tend to explain their reasoning:

# Without format instruction - verbose response
chat$chat("Classify this as billing/technical/account: I can't log in")
# "This appears to be an account-related issue because..."

# With format instruction - clean response
chat$chat("Classify as billing/technical/account. One word only: I can't log in")
# "account"

Clean responses are easier to process programmatically.

Structured Classification

Using extract_data for reliable output

For production use, extract_data() guarantees structured output:

# Define the allowed categories
category_type <- type_enum(
  values = c("billing", "technical", "account", "other"),
  description = "Support ticket category"
)

The type_enum() ensures the response is one of the allowed values.

Extract the category

chat <- chat_claude()

ticket <- "Why was I charged twice this month? Please refund the extra charge."

category <- chat$extract_data(ticket, type = category_type)

category
# "billing"

This approach:

Guarantees valid output (no unexpected categories)
Returns clean data (no extra text)
Works reliably in automated pipelines

Multi-label Classification

When text belongs to multiple categories

Some text fits multiple categories. Use an array type:

categories_type <- type_array(
  items = type_enum(
    values = c("urgent", "billing", "technical", "account", "feedback"),
    description = "Applicable category"
  ),
  description = "All categories that apply to this ticket"
)

Classify with multiple labels

chat <- chat_claude()

ticket <- "URGENT: My payment failed and now I'm locked out of my account!"

labels <- chat$extract_data(ticket, type = categories_type)

labels
# ["urgent", "billing", "account"]

The ticket is urgent, involves billing (payment failed), and account access.

Classification with Confidence

Get confidence scores

Sometimes you want to know how confident the model is:

classification_type <- type_object(
  category = type_enum(
    values = c("billing", "technical", "account", "other"),
    description = "Primary category"
  ),
  confidence = type_number("Confidence score from 0 to 1"),
  reasoning = type_string("Brief explanation for the classification")
)

Extract classification with confidence

chat <- chat_claude()

ticket <- "The app crashes when I try to upload files larger than 10MB"

result <- chat$extract_data(ticket, type = classification_type)

result$category
# "technical"

result$confidence
# 0.95

result$reasoning
# "This describes a software bug related to file uploads"

Low confidence scores indicate ambiguous cases that may need human review.

Batch Classification

Create a reusable classifier

Wrap classification in a function for reuse:

classify_ticket <- function(text, chat = NULL) {
  if (is.null(chat)) {
    chat <- chat_claude()
  }

  category_type <- type_enum(
    values = c("billing", "technical", "account", "other"),
    description = "Ticket category"
  )

  chat$extract_data(text, type = category_type)
}

Classify multiple texts

Use purrr’s map() to process a vector of texts:

library(purrr)

tickets <- c(
 "I was charged twice for my subscription",
 "The export feature isn't working",
 "How do I change my email address?",
 "Your product is amazing, thank you!"
)

# Add delay to respect rate limits
categories <- map_chr(tickets, \(ticket) {
 Sys.sleep(0.5)
 classify_ticket(ticket)
})

categories
# ["billing", "technical", "account", "other"]

The Sys.sleep(0.5) prevents hitting API rate limits.

Create a classified data frame

tibble(
 ticket = tickets,
 category = categories
)

ticket	category
I was charged twice…	billing
The export feature…	technical
How do I change…	account
Your product is…	other

Hierarchical Classification

Two-level categories

For complex taxonomies, classify in stages:

# First level: main category
main_category <- type_enum(
  values = c("technical", "billing", "account"),
  description = "Main category"
)

# Second level: subcategories for technical issues
technical_subcategory <- type_enum(
  values = c("bug", "feature_request", "how_to", "performance"),
  description = "Technical subcategory"
)

Two-stage classification

classify_hierarchical <- function(text) {
  chat <- chat_claude()

  # Stage 1: Main category
  main <- chat$extract_data(text, type = main_category)

  # Stage 2: Subcategory (only for technical)
  sub <- if (main == "technical") {
    chat$extract_data(text, type = technical_subcategory)
  } else {
    NA
  }

  list(category = main, subcategory = sub)
}

result <- classify_hierarchical("The dashboard loads very slowly")
# $category: "technical"
# $subcategory: "performance"

Custom Categories

Define your own taxonomy

LLMs work with any categories you define:

# Document types
doc_type <- type_enum(
  values = c("contract", "invoice", "report", "memo", "other"),
  description = "Document type"
)

# Email intents
email_intent <- type_enum(
  values = c("inquiry", "complaint", "request", "thank_you", "spam"),
  description = "Email intent"
)

# News topics
news_topic <- type_enum(
  values = c("politics", "business", "technology", "sports", "entertainment"),
  description = "News article topic"
)

Use with any text

chat <- chat_claude()

article <- "Apple announced record quarterly earnings driven by iPhone sales in Asia."

topic <- chat$extract_data(article, type = news_topic)
# "business"

Improving Accuracy

Add context to the system prompt

Provide domain context for better classification:

chat <- chat_claude(
  system_prompt = "You are a customer support classifier for a SaaS company.

  Category definitions:
  - billing: payments, charges, refunds, subscriptions, pricing
  - technical: bugs, errors, features not working, integrations
  - account: login, password, profile, settings, permissions
  - other: feedback, partnerships, general questions"
)

category_type <- type_enum(
  values = c("billing", "technical", "account", "other"),
  description = "Ticket category"
)

ticket <- "Can you add dark mode to the app?"

chat$extract_data(ticket, type = category_type)
# "other" (feature request, not a bug)

Provide examples (few-shot)

Include examples in the prompt for tricky cases:

chat <- chat_claude(
  system_prompt = "Classify support tickets. Examples:

  'Charge me yearly instead of monthly' -> billing
  'Button doesn't work on mobile' -> technical
  'Reset my 2FA' -> account
  'Love your product!' -> other"
)

Error Handling

Handle API failures gracefully

safe_classify <- function(text) {
  tryCatch({
    chat <- chat_claude()
    category_type <- type_enum(
      values = c("billing", "technical", "account", "other"),
      description = "Category"
    )
    chat$extract_data(text, type = category_type)
  }, error = function(e) {
    warning("Classification failed: ", e$message)
    NA_character_
  })
}

# Use with map
results <- map_chr(tickets, safe_classify)

# Filter out failures
valid_results <- results[!is.na(results)]

Local Classification

Use Ollama for free, private classification

For sensitive data or high volume, use local models:

chat <- chat_ollama(model = "llama3.2")

category_type <- type_enum(
  values = c("billing", "technical", "account", "other"),
  description = "Category"
)

chat$extract_data("I need a refund", type = category_type)
# "billing"

See Local LLMs with Ollama for setup.

Note: Local models may be less accurate than cloud APIs for nuanced classification.

Performance Tips

Batch similar classifications

Process items in batches when possible:

# Instead of classifying one at a time
# Classify multiple items with context

batch_classify <- function(texts) {
  chat <- chat_claude()

  # Build numbered list
  numbered <- paste(seq_along(texts), texts, sep = ". ", collapse = "\n")

  prompt <- paste(
    "Classify each item as billing/technical/account/other.",
    "Return a JSON array of categories in order.",
    "\n\n", numbered
  )

  # Parse JSON response
  jsonlite::fromJSON(chat$chat(prompt))
}

Use cheaper models for simple tasks

# Use Haiku for simple classification
chat <- chat_claude(model = "claude-haiku-4-5")

# Use Opus only for complex multi-label tasks
chat <- chat_claude(model = "claude-opus-4-6")

Summary

Task	Code
Simple classification	`chat$chat("Classify as X/Y/Z: text")`
Structured output	`chat$extract_data(text, type_enum(...))`
Multi-label	`type_array(items = type_enum(...))`
With confidence	`type_object(category = ..., confidence = ...)`
Batch processing	`map_chr(texts, classify_fn)`

Key points:

Use type_enum() to guarantee valid categories
Add system prompts with category definitions for accuracy
Use Sys.sleep() in batch processing for rate limits
Consider local models for sensitive or high-volume data

Sources

How to Extract Structured Data with LLMs in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

LLMs excel at extracting structured information from unstructured text. Instead of writing complex regex patterns, you can describe what you want and get clean, structured output.

This tutorial uses the ellmer package. You can use any provider: Claude, OpenAI, or local models with Ollama.

Use cases: - Extract names, dates, addresses from text - Parse product information from descriptions - Convert free-text survey responses to categories - Extract entities from documents - Clean and standardize messy data

Getting Started

library(ellmer)
library(tidyverse)

Basic Extraction

Define a schema

Tell the LLM what structure you expect:

person_schema <- type_object(
  name = type_string("Person's full name"),
  age = type_integer("Person's age in years"),
  email = type_string("Email address if mentioned")
)

Extract from text

chat <- chat_claude()

result <- chat$extract_data(
  "John Smith is 35 years old. You can reach him at john@email.com",
  type = person_schema
)

Access the results

result$name   # "John Smith"
result$age    # 35
result$email  # "john@email.com"

Handle missing data

result <- chat$extract_data(
  "Sarah is 28 years old.",
  type = person_schema
)

result
# $name: "Sarah"
# $age: 28
# $email: NULL  # Not mentioned in text

Type Definitions

Available types

# String
type_string("description of the field")

# Integer
type_integer("description")

# Number (float)
type_number("description")

# Boolean
type_boolean("description")

# Enum (predefined options)
type_enum(
  values = c("positive", "negative", "neutral"),
  description = "Sentiment classification"
)

# Array (list of items)
type_array(
  items = type_string("individual item description"),
  description = "A list of items"
)

# Object (nested structure)
type_object(
  field1 = type_string("..."),
  field2 = type_integer("...")
)

Practical Examples

Extract product information

product_schema <- type_object(
  name = type_string("Product name"),
  price = type_number("Price in dollars"),
  currency = type_string("Currency code"),
  features = type_array(
    items = type_string("A product feature")
  )
)

text <- "The iPhone 15 Pro costs $999. Features include titanium design,
A17 chip, 48MP camera, and USB-C port."

chat <- chat_claude()
product <- chat$extract_data(text, type = product_schema)

product
# $name: "iPhone 15 Pro"
# $price: 999
# $currency: "USD"
# $features: ["titanium design", "A17 chip", "48MP camera", "USB-C port"]

Classify sentiment

sentiment_schema <- type_object(
  sentiment = type_enum(
    values = c("positive", "negative", "neutral"),
    description = "Overall sentiment"
  ),
  confidence = type_number("Confidence score from 0 to 1"),
  key_phrases = type_array(
    items = type_string("Key phrase indicating sentiment")
  )
)

review <- "Absolutely love this product! Best purchase I've made this year.
The quality is outstanding and shipping was super fast."

chat <- chat_claude()
result <- chat$extract_data(review, type = sentiment_schema)

result
# $sentiment: "positive"
# $confidence: 0.95
# $key_phrases: ["Absolutely love", "Best purchase", "outstanding", "super fast"]

Parse contact information

contact_schema <- type_object(
  name = type_string("Full name"),
  phone = type_string("Phone number"),
  email = type_string("Email address"),
  address = type_object(
    street = type_string("Street address"),
    city = type_string("City"),
    state = type_string("State"),
    zip = type_string("ZIP code")
  )
)

text <- "Contact Jane Doe at (555) 123-4567 or jane.doe@company.com.
Her office is at 123 Main Street, San Francisco, CA 94102."

chat <- chat_claude()
contact <- chat$extract_data(text, type = contact_schema)

Extract dates and events

event_schema <- type_object(
  event_name = type_string("Name of the event"),
  date = type_string("Date in YYYY-MM-DD format"),
  location = type_string("Event location"),
  description = type_string("Brief description")
)

text <- "Join us for the R Users Meetup on March 15th, 2024 at the
Downtown Conference Center. We'll discuss data visualization techniques."

chat <- chat_claude()
event <- chat$extract_data(text, type = event_schema)

event
# $event_name: "R Users Meetup"
# $date: "2024-03-15"
# $location: "Downtown Conference Center"
# $description: "Discussion about data visualization techniques"

Extracting Multiple Items

Extract array of objects

person_schema <- type_object(
  name = type_string("Person's name"),
  role = type_string("Person's role or title")
)

people_schema <- type_array(
  items = person_schema,
  description = "List of people mentioned"
)

text <- "The meeting included CEO John Smith, CTO Sarah Johnson,
and CFO Michael Brown. They discussed Q4 results."

chat <- chat_claude()
people <- chat$extract_data(text, type = people_schema)

people
# [[1]] $name: "John Smith", $role: "CEO"
# [[2]] $name: "Sarah Johnson", $role: "CTO"
# [[3]] $name: "Michael Brown", $role: "CFO"

Convert to data frame

# Extract as tibble
people_df <- tibble(
  name = map_chr(people, "name"),
  role = map_chr(people, "role")
)

people_df

Batch Processing

Process multiple texts

library(purrr)

reviews <- c(
  "Great product, love it!",
  "Terrible quality, very disappointed",
  "It's okay, nothing special",
  "Best purchase ever, highly recommend"
)

sentiment_schema <- type_enum(
  values = c("positive", "negative", "neutral"),
  description = "Sentiment"
)

extract_sentiment <- function(text) {
  chat <- chat_claude()
  Sys.sleep(0.5)  # Rate limiting
  chat$extract_data(text, type = sentiment_schema)
}

sentiments <- map_chr(reviews, extract_sentiment)

tibble(
  review = reviews,
  sentiment = sentiments
)

Process data frame column

df <- tibble(
  id = 1:3,
  description = c(
    "John Smith, age 30, engineer",
    "Jane Doe, age 25, designer",
    "Bob Brown, age 45, manager"
  )
)

person_schema <- type_object(
  name = type_string("Name"),
  age = type_integer("Age"),
  job = type_string("Job title")
)

df_extracted <- df |>
  mutate(
    extracted = map(description, \(text) {
      chat <- chat_claude()
      Sys.sleep(0.5)
      chat$extract_data(text, type = person_schema)
    }),
    name = map_chr(extracted, "name"),
    age = map_int(extracted, "age"),
    job = map_chr(extracted, "job")
  ) |>
  select(-extracted)

df_extracted

Advanced Patterns

Extraction with instructions

chat <- chat_claude(
  system_prompt = "Extract information exactly as specified.
  If information is unclear, make your best inference.
  Use NULL for genuinely missing data."
)

result <- chat$extract_data(text, type = schema)

Validate extracted data

extract_and_validate <- function(text, schema, validation_fn) {
  chat <- chat_claude()
  result <- chat$extract_data(text, type = schema)

  if (!validation_fn(result)) {
    warning("Extraction may be incomplete or invalid")
  }

  result
}

# Example validation
validate_person <- function(person) {
  !is.null(person$name) && !is.null(person$age)
}

result <- extract_and_validate(
  "Some text",
  person_schema,
  validate_person
)

Combine extraction with classification

ticket_schema <- type_object(
  category = type_enum(
    values = c("billing", "technical", "account", "other"),
    description = "Support ticket category"
  ),
  priority = type_enum(
    values = c("low", "medium", "high", "urgent"),
    description = "Priority level"
  ),
  summary = type_string("One-sentence summary"),
  entities = type_object(
    account_id = type_string("Account ID if mentioned"),
    error_code = type_string("Error code if mentioned")
  )
)

ticket <- "Hi, I can't log into my account #12345. Getting error E401.
This is urgent as I need to complete a transaction today!"

chat <- chat_claude()
parsed_ticket <- chat$extract_data(ticket, type = ticket_schema)

Error Handling

safe_extract <- function(text, schema) {
  tryCatch({
    chat <- chat_claude()
    chat$extract_data(text, type = schema)
  }, error = function(e) {
    warning("Extraction failed: ", e$message)
    NULL
  })
}

# Use with map for batch processing
results <- map(texts, \(t) safe_extract(t, schema))

# Filter out failures
valid_results <- compact(results)  # Remove NULLs

Local LLM Extraction

Use Ollama for free, private extraction:

# Works the same way with local models
chat <- chat_ollama(model = "llama3.2")

result <- chat$extract_data(
  "John Smith, 35 years old, john@email.com",
  type = person_schema
)

Note: Local models may be less accurate for complex schemas. Test thoroughly.

Common Mistakes

1. Schema too complex

# Too many nested levels can confuse the model
# Break into simpler extractions if needed

2. Ambiguous field descriptions

# Bad
type_string("date")

# Good
type_string("Event date in YYYY-MM-DD format")

3. Not handling NULL values

# Always check for NULLs
result$field %||% "default_value"

# Or use map with default
map_chr(results, "field", .default = NA_character_)

4. Forgetting rate limits in batches

# Always add delays
map(texts, \(t) {
  Sys.sleep(0.5)  # Important!
  extract(t)
})

Summary

Task	Code
Define string field	`type_string("description")`
Define number field	`type_number("description")`
Define enum field	`type_enum(values = c(...))`
Define array	`type_array(items = type_*())`
Define object	`type_object(field = type_*())`
Extract data	`chat$extract_data(text, type)`

Define schemas with type_*() functions
Use clear field descriptions
Handle NULL values for missing data
Add delays when batch processing
Validate extracted data when reliability is important

Sources

How to Run Local LLMs in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

Running LLMs locally gives you: - No API costs - completely free after setup - Privacy - data never leaves your machine - Offline access - works without internet - No rate limits - unlimited requests

Ollama makes local LLMs easy by handling model downloads, memory management, and providing a simple API. We’ll use the ellmer package to connect R to Ollama.

Prefer cloud APIs? See OpenAI or Claude for more powerful models.

Getting Started

Install Ollama

Download and install from ollama.com:

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from ollama.com

Start Ollama

# Start the Ollama server
ollama serve

Download a model

# Download Llama 3 (8B parameters, ~4GB)
ollama pull llama3.2

# Download smaller model (faster)
ollama pull llama3.2:1b

# Download Mistral
ollama pull mistral

Using Ollama in R

With ellmer (recommended)

install.packages("ellmer")
library(ellmer)

# Connect to local Ollama
chat <- chat_ollama(model = "llama3.2")
chat$chat("What is R programming?")

With ollamar package

install.packages("ollamar")
library(ollamar)

# Generate text
generate("llama3.2", "Explain data frames in R")

# Chat format
chat(
  model = "llama3.2",
  messages = list(
    list(role = "user", content = "What is ggplot2?")
  )
)

Available Models

Recommended for R coding

Model	Size	RAM Needed	Best For
`llama3.2:1b`	1.3GB	4GB	Fast responses, simple tasks
`llama3.2`	4.7GB	8GB	Balanced, good for coding
`codellama`	3.8GB	8GB	Code generation
`mistral`	4.1GB	8GB	General tasks
`mixtral`	26GB	48GB	Best quality, needs lots of RAM

Download models

# Code-focused
ollama pull codellama

# General purpose
ollama pull llama3.2
ollama pull mistral

# Smaller/faster
ollama pull llama3.2:1b
ollama pull phi3

List installed models

library(ollamar)
list_models()

Basic Usage with ellmer

Simple chat

library(ellmer)

chat <- chat_ollama(model = "llama3.2")
chat$chat("How do I read a CSV file in R?")

With system prompt

chat <- chat_ollama(
  model = "llama3.2",
  system_prompt = "You are an R programming expert. Provide concise answers with code examples."
)

chat$chat("How do I calculate the mean by group?")

Multi-turn conversation

chat <- chat_ollama(model = "llama3.2")

chat$chat("I have a dataset with customer purchase data.")
chat$chat("How would I find the top 10 customers by total spend?")
chat$chat("Now show me how to visualize this.")

Practical Examples

Generate R code

chat <- chat_ollama(
  model = "codellama",
  system_prompt = "Return only R code. No explanations."
)

code <- chat$chat("
Write a function that:
1. Takes a data frame
2. Finds numeric columns
3. Scales them to 0-1 range
4. Returns the modified data frame
")

cat(code)

Analyze text data

classify_sentiment <- function(text) {
  chat <- chat_ollama(
    model = "llama3.2",
    system_prompt = "Classify sentiment as positive, negative, or neutral. Reply with one word only."
  )
  chat$chat(text)
}

reviews <- c(
  "This product is amazing!",
  "Terrible quality, very disappointed",
  "It works as expected"
)

sapply(reviews, classify_sentiment)

Explain code

chat <- chat_ollama(model = "llama3.2")

code <- "
mtcars |>
  filter(mpg > 20) |>
  group_by(cyl) |>
  summarise(mean_hp = mean(hp))
"

chat$chat(paste("Explain this R code:", code))

Using ollamar Directly

Generate text

library(ollamar)

result <- generate(
  model = "llama3.2",
  prompt = "Write R code to create a bar chart"
)

result$response

Chat with history

messages <- list(
  list(role = "user", content = "What is dplyr?")
)

response <- chat(model = "llama3.2", messages = messages)

# Continue conversation
messages <- c(messages, list(
  list(role = "assistant", content = response$message$content),
  list(role = "user", content = "Show me an example of filter()")
))

response2 <- chat(model = "llama3.2", messages = messages)

Embeddings for semantic search

# Generate embeddings
embedding <- embeddings(
  model = "llama3.2",
  input = "How to handle missing values in R"
)

# Use for similarity search, clustering, etc.
embedding$embedding

Performance Tips

Choose the right model size

# Fast but less capable (good for simple tasks)
chat <- chat_ollama(model = "llama3.2:1b")

# Balanced (good for most tasks)
chat <- chat_ollama(model = "llama3.2")

# Best quality (slower, needs more RAM)
chat <- chat_ollama(model = "mixtral")

Limit context length

# Shorter context = faster responses
chat <- chat_ollama(
  model = "llama3.2",
  api_args = list(num_ctx = 2048)  # Default is 4096
)

GPU acceleration

# Ollama automatically uses GPU if available
# Check GPU usage:
ollama ps

Batch Processing

library(purrr)

texts <- c("Text 1", "Text 2", "Text 3")

process_local <- function(texts, prompt_template) {
  map_chr(texts, \(text) {
    chat <- chat_ollama(model = "llama3.2")
    chat$chat(sprintf(prompt_template, text))
  })
}

summaries <- process_local(
  texts,
  "Summarize in one sentence: %s"
)

Comparing Local vs Cloud

Feature	Local (Ollama)	Cloud (OpenAI/Claude)
Cost	Free	Pay per token
Privacy	Complete	Data sent to servers
Speed	Depends on hardware	Generally fast
Quality	Good (varies by model)	Best available
Offline	Yes	No
Rate limits	None	Yes

When to use local

Privacy-sensitive data
High volume, low budget
Offline requirements
Learning/experimentation

When to use cloud

Need best quality
Don’t have GPU
Production applications
Complex reasoning tasks

Troubleshooting

Ollama not running

# Check if Ollama is running
tryCatch({
  chat <- chat_ollama(model = "llama3.2")
  chat$chat("test")
}, error = function(e) {
  message("Start Ollama with: ollama serve")
})

Model not found

# List available models
ollama list

# Pull missing model
ollama pull llama3.2

Out of memory

# Use smaller model
ollama pull llama3.2:1b

# Or reduce context
# Set num_ctx to lower value in R

Slow responses

# Use smaller model
chat <- chat_ollama(model = "llama3.2:1b")

# Reduce max tokens
chat <- chat_ollama(
  model = "llama3.2",
  api_args = list(num_predict = 100)
)

Common Mistakes

1. Forgetting to start Ollama server

# Must run this first
ollama serve

2. Using model that’s not downloaded

# Check what's installed
ollama list

# Download if needed
ollama pull llama3.2

3. Expecting cloud-level quality

# Local models are good but not as capable as GPT-4 or Claude
# Adjust expectations and prompts accordingly

4. Not providing good prompts

# Be specific with local models
# They need clearer instructions than cloud models

# Too vague
chat$chat("help with data")

# Better
chat$chat("Write R code to calculate the mean of the 'price' column in a data frame called 'sales'")

Summary

Task	Code
Start Ollama	`ollama serve` (terminal)
Download model	`ollama pull llama3.2` (terminal)
Chat with ellmer	`chat_ollama(model = "llama3.2")`
Generate text	`ollamar::generate(model, prompt)`
List models	`ollamar::list_models()`

Install Ollama from ollama.com
Download models with ollama pull
Use chat_ollama() from ellmer for easy integration
Smaller models (1b, 3b) are faster but less capable
Local LLMs are free, private, and work offline

Sources

How to Use Claude API in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

Claude is Anthropic’s flagship AI model and one of the most popular coding assistants available. It excels at writing, understanding, and debugging R code. The ellmer package provides a tidyverse-friendly way to use Claude’s API in R.

Alternatives: See also OpenAI API or run models locally with Ollama for free.

What you’ll learn: - Set up Claude API access - Use ellmer’s chat_claude() function - Create multi-turn conversations - Use tool calling for R functions - Extract structured data

Getting Started

Install ellmer

install.packages("ellmer")
library(ellmer)

Get your API key

Create an account at console.anthropic.com
Go to API Keys section
Create a new key
Add billing information (required for API access)

Note: A Claude Pro subscription does NOT include API access. You need a separate developer account.

Set your API key

# Option 1: Set for current session
Sys.setenv(ANTHROPIC_API_KEY = "sk-ant-your-key-here")

# Option 2: Add to .Renviron (recommended)
usethis::edit_r_environ()
# Add: ANTHROPIC_API_KEY=sk-ant-your-key-here

Basic Chat

Create a chat session

library(ellmer)

chat <- chat_claude()
chat$chat("What is R programming?")

Single question

chat <- chat_claude()
response <- chat$chat("Explain what a data frame is in R.")
response

Specify model

# Use specific Claude model
chat <- chat_claude(model = "claude-opus-4-6")

Available Models

Latest Models (Claude 4.6)

Model	Best For	Context	Max Output
`claude-opus-4-6`	Agents, complex coding	1M tokens	128k tokens
`claude-sonnet-4-6`	Speed + intelligence	1M tokens	64k tokens
`claude-haiku-4-5`	Fastest, budget	200k tokens	64k tokens

Legacy Models (still available)

Model	Context
`claude-opus-4-5`	200k tokens
`claude-sonnet-4-5`	200k tokens
`claude-sonnet-4-0`	200k tokens

# Claude Opus 4.6 (most capable)
chat <- chat_claude(model = "claude-opus-4-6")

# Claude Sonnet 4.6 (balanced, recommended)
chat <- chat_claude(model = "claude-sonnet-4-6")

# Claude Haiku 4.5 (fastest, cheapest)
chat <- chat_claude(model = "claude-haiku-4-5")

Note: Claude 4.6 models support extended thinking and have 1M token context windows.

System Prompts

Control Claude’s behavior:

chat <- chat_claude(
  system_prompt = "You are an expert R programmer. Always provide working code examples. Be concise."
)

chat$chat("How do I calculate the mean of a column?")

Specialized assistants

# Data analysis assistant
analyst <- chat_claude(
  system_prompt = "You are a data analyst expert in R and the tidyverse.
  When asked questions, provide clear explanations with code examples using dplyr and ggplot2."
)

analyst$chat("How do I find outliers in my data?")

Multi-turn Conversations

ellmer maintains conversation history automatically:

chat <- chat_claude()

# First message
chat$chat("I have a dataset of customer purchases.")

# Follow-up (Claude remembers context)
chat$chat("How would I calculate total spending per customer?")

# Another follow-up
chat$chat("Now how do I visualize this?")

View conversation history

# See all turns
chat$get_turns()

Practical Examples

Generate R code

chat <- chat_claude(
  system_prompt = "You are an R expert. Return only executable R code with comments. No explanations outside code."
)

code <- chat$chat("
Create a function that:
1. Takes a data frame and column name
2. Removes outliers (values beyond 1.5*IQR)
3. Returns the cleaned data frame
")

cat(code)

Explain existing code

chat <- chat_claude()

code_to_explain <- "
mtcars |>
  group_by(cyl) |>
  summarise(across(where(is.numeric), mean)) |>
  pivot_longer(-cyl)
"

chat$chat(paste("Explain this R code step by step:", code_to_explain))

Debug errors

chat <- chat_claude(
  system_prompt = "You are an R debugging expert. When shown errors, explain the cause and provide a fix."
)

error_message <- "Error in select(df, name) : object 'name' not found"

chat$chat(paste("I got this error:", error_message,
                "My code was: df |> select(name)"))

Analyze data descriptions

chat <- chat_claude()

chat$chat("
I have a dataset with these columns:
- customer_id (integer)
- purchase_date (date)
- amount (numeric)
- category (character: 'electronics', 'clothing', 'food')

Suggest 5 interesting analyses I could perform.
")

Tool Calling

Tool calling lets Claude execute R functions to get real data. This is powerful for building AI agents.

Define your function

First, create a regular R function:

get_weather <- function(city) {
  # In practice, this would call a weather API
  paste("Weather in", city, ": 72°F, sunny")
}

Register it as a tool

Tell Claude about the function:

chat <- chat_claude()

chat$register_tool(
  name = "get_weather",
  description = "Get current weather for a city",
  arguments = list(
    city = tool_arg("string", "The city name")
  ),
  func = get_weather
)

Claude calls it automatically

When you ask about weather, Claude recognizes it should use the tool:

chat$chat("What's the weather in New York?")
# Claude calls get_weather("New York") and returns the result

Tool for data analysis

Create a function that analyzes data:

analyze_data <- function(column_name) {
  data <- mtcars[[column_name]]
  list(mean = mean(data), sd = sd(data), min = min(data), max = max(data))
}

chat <- chat_claude()

chat$register_tool(
  name = "analyze_data",
  description = "Get summary statistics for a column in mtcars",
  arguments = list(column_name = tool_arg("string", "Column name")),
  func = analyze_data
)

chat$chat("What are the statistics for the mpg column?")

Structured Output

Extract data in a specific format instead of free text. This is useful for parsing reviews, extracting entities, or classifying content. For more examples, see How to Extract Structured Data with LLMs.

Define a schema

Specify the structure you want:

review_schema <- type_object(
  sentiment = type_string("positive, negative, or neutral"),
  confidence = type_number("confidence score 0-1"),
  summary = type_string("one sentence summary")
)

Extract structured data

Pass text and schema to get clean output:

chat <- chat_claude()

result <- chat$extract_data(
  "This product exceeded my expectations! Great quality and fast shipping.",
  type = review_schema
)

Result is a list

result$sentiment
# "positive"

result$confidence
# 0.95

Streaming Responses

For long responses, stream output:

chat <- chat_claude()

# Stream response (prints as it generates)
chat$stream("Write a detailed guide to ggplot2 themes.")

Error Handling

safe_chat <- function(prompt) {
  tryCatch({
    chat <- chat_claude()
    chat$chat(prompt)
  }, error = function(e) {
    message("API Error: ", e$message)
    NA
  })
}

result <- safe_chat("What is 2+2?")

Common errors

Error	Cause	Solution
401 Unauthorized	Invalid API key	Check ANTHROPIC_API_KEY
429 Rate limit	Too many requests	Add delays
Credit balance	No funds	Add billing

Cost Management

Token usage

# Check usage after chat
chat <- chat_claude()
chat$chat("Hello!")

# View token counts
chat$get_turns()

Use appropriate models

# Use Haiku for simple tasks (cheapest)
simple_chat <- chat_claude(model = "claude-haiku-4-5")

# Use Sonnet for coding (balanced, recommended)
code_chat <- chat_claude(model = "claude-sonnet-4-6")

# Use Opus only for complex reasoning (most capable)
complex_chat <- chat_claude(model = "claude-opus-4-6")

Batch Processing

Process multiple texts with purrr’s map() functions.

Create a processing function

classify_sentiment <- function(text) {
  chat <- chat_claude(
    system_prompt = "Classify as positive/negative/neutral. One word only."
  )
  Sys.sleep(0.5)  # Rate limiting - important!
  chat$chat(text)
}

Apply to multiple texts

library(purrr)

texts <- c("Great product!", "Terrible service", "It's okay I guess")

results <- map_chr(texts, classify_sentiment)
# "positive", "negative", "neutral"

The Sys.sleep(0.5) prevents hitting rate limits when processing many items.

Common Mistakes

1. Using Claude Pro subscription for API

# Claude Pro (claude.ai) is NOT the same as API access
# You need a developer account at console.anthropic.com

2. Forgetting conversation state

# Each chat_claude() creates a NEW conversation
chat1 <- chat_claude()
chat1$chat("My name is Alice")

chat2 <- chat_claude()  # New conversation!
chat2$chat("What's my name?")  # Claude doesn't know

# Reuse same chat object for context
chat1$chat("What's my name?")  # "Alice"

3. Not setting appropriate system prompts

# Generic (may give verbose responses)
chat <- chat_claude()

# Better (focused responses)
chat <- chat_claude(
  system_prompt = "Be concise. Respond in 2-3 sentences max."
)

Summary

Task	Code
Basic chat	`chat <- chat_claude(); chat$chat("Hi")`
Set API key	`Sys.setenv(ANTHROPIC_API_KEY = "key")`
System prompt	`chat_claude(system_prompt = "...")`
Specific model	`chat_claude(model = "claude-opus-4-6")`
Tool calling	`chat$register_tool(...)`
Structured output	`chat$extract_data(text, type)`

Claude Sonnet is best for R code generation
Use system prompts to control response style
Reuse chat objects to maintain conversation context
Add delays when processing multiple items

Sources

How to Use ellmer in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

ellmer is Posit’s official R package for working with Large Language Models. It provides a consistent, tidyverse-friendly interface that works with multiple LLM providers.

Key features: - Consistent API across providers (OpenAI, Claude, Ollama, etc.) - Streaming output support - Tool/function calling - Structured data extraction - Conversation history management

Getting Started

install.packages("ellmer")
library(ellmer)

Supported Providers

Provider	Function	API Key Variable
OpenAI	`chat_openai()`	`OPENAI_API_KEY`
Anthropic (Claude)	`chat_claude()`	`ANTHROPIC_API_KEY`
Google (Gemini)	`chat_gemini()`	`GOOGLE_API_KEY`
Ollama (Local)	`chat_ollama()`	None (local)
Azure OpenAI	`chat_azure()`	`AZURE_OPENAI_API_KEY`
AWS Bedrock	`chat_bedrock()`	AWS credentials

Basic Usage

Create a chat

# OpenAI
chat <- chat_openai()

# Claude
chat <- chat_claude()

# Local (Ollama)
chat <- chat_ollama()

Send a message

chat <- chat_openai()
response <- chat$chat("What is the tidyverse?")
response

Multi-turn conversation

chat <- chat_claude()

chat$chat("I'm analyzing sales data.")
chat$chat("How should I handle missing values?")
chat$chat("Show me code to do that.")

System Prompts

Control assistant behavior:

chat <- chat_openai(
  system_prompt = "You are an R programming expert. Always provide working code examples using tidyverse conventions."
)

chat$chat("How do I join two data frames?")

Role-specific assistants

# Data analyst
analyst <- chat_claude(
  system_prompt = "You are a senior data analyst. Explain concepts clearly and suggest best practices for data analysis in R."
)

# Code reviewer
reviewer <- chat_openai(
  system_prompt = "You are a code reviewer. Analyze R code for bugs, inefficiencies, and style issues. Be constructive."
)

# Statistics tutor
tutor <- chat_claude(
  system_prompt = "You are a statistics tutor. Explain statistical concepts in simple terms with R examples."
)

Switching Between Providers

Same code works across providers:

# Function that works with any provider
analyze_with_llm <- function(chat, data_description) {
  chat$chat(paste(
    "I have this data:",
    data_description,
    "Suggest 3 analyses I could perform."
  ))
}

# Use with different providers
openai_chat <- chat_openai()
claude_chat <- chat_claude()
local_chat <- chat_ollama()

# Same function, different backends
analyze_with_llm(openai_chat, "Customer purchase history")
analyze_with_llm(claude_chat, "Customer purchase history")
analyze_with_llm(local_chat, "Customer purchase history")

Streaming Output

For long responses, stream as they generate:

chat <- chat_claude()

# Prints incrementally as response generates
chat$stream("Write a comprehensive guide to data visualization in R.")

Tool Calling

Tool calling lets LLMs execute R functions to get real data.

Define a function

calculate_stats <- function(numbers) {
  list(mean = mean(numbers), sd = sd(numbers), median = median(numbers))
}

Register it as a tool

chat <- chat_claude()

chat$register_tool(
  name = "calculate_stats",
  description = "Calculate summary statistics for a vector of numbers",
  arguments = list(
    numbers = tool_arg("array", "A vector of numbers", items = type_number())
  ),
  func = calculate_stats
)

# LLM can now call this function
chat$chat("What are the statistics for the numbers 1, 5, 3, 9, 2, 7?")

Multiple tools

# Tool 1: Read data
read_data <- function(filename) {
  if (file.exists(filename)) {
    head(read.csv(filename), 5)
  } else {
    "File not found"
  }
}

# Tool 2: Plot data
create_plot <- function(x_col, y_col) {
  paste("Creating plot of", y_col, "vs", x_col)
}

chat <- chat_openai()

chat$register_tool(
  name = "read_data",
  description = "Read first 5 rows of a CSV file",
  arguments = list(filename = tool_arg("string", "Path to CSV file")),
  func = read_data
)

chat$register_tool(
  name = "create_plot",
  description = "Create a scatter plot",
  arguments = list(
    x_col = tool_arg("string", "X axis column"),
    y_col = tool_arg("string", "Y axis column")
  ),
  func = create_plot
)

chat$chat("Read data.csv and plot mpg vs wt")

Structured Data Extraction

Extract data in specific formats. For a deep dive, see How to Extract Structured Data with LLMs.

Simple extraction

Define a schema for the data you want:

person_type <- type_object(
  name = type_string("Person's full name"),
  age = type_integer("Person's age"),
  occupation = type_string("Person's job")
)

Extract structured data from text:

chat <- chat_claude()

result <- chat$extract_data(
  "John Smith is a 35-year-old software engineer from Seattle.",
  type = person_type
)

result$name  # "John Smith"
result$age   # 35
# $name: "John Smith"
# $age: 35
# $occupation: "software engineer"

Extract arrays

# Schema for multiple items
products_type <- type_array(
  items = type_object(
    name = type_string("Product name"),
    price = type_number("Price in dollars"),
    category = type_string("Product category")
  )
)

text <- "We have laptops at $999, headphones for $199, and keyboards at $79."

chat <- chat_openai()
products <- chat$extract_data(text, type = products_type)

Classification

sentiment_type <- type_enum(
  values = c("positive", "negative", "neutral"),
  description = "The sentiment of the text"
)

chat <- chat_claude()
sentiment <- chat$extract_data(
  "This product is amazing!",
  type = sentiment_type
)
# "positive"

Conversation Management

View history

chat <- chat_claude()
chat$chat("Hello!")
chat$chat("What's 2+2?")

# Get all turns
turns <- chat$get_turns()

Clear history

# Start fresh
chat$clear()

Save and restore

# Save conversation
turns <- chat$get_turns()
saveRDS(turns, "conversation.rds")

# Restore later
saved_turns <- readRDS("conversation.rds")
chat <- chat_claude(turns = saved_turns)

Practical Examples

Code generation pipeline

generate_and_run <- function(task) {
  chat <- chat_claude(
    system_prompt = "Return only valid R code. No explanations. No markdown."
  )

  code <- chat$chat(task)

  # Try to run the code
  tryCatch({
    result <- eval(parse(text = code))
    list(code = code, result = result, success = TRUE)
  }, error = function(e) {
    list(code = code, error = e$message, success = FALSE)
  })
}

generate_and_run("Calculate the mean of mtcars$mpg")

Data summarization

summarize_data <- function(df, chat = chat_claude()) {
  description <- paste(
    "Dataset with", nrow(df), "rows and", ncol(df), "columns.",
    "Columns:", paste(names(df), collapse = ", "),
    "First few values:", capture.output(head(df, 3))
  )

  chat$chat(paste(
    "Here's a dataset:",
    description,
    "Provide a brief summary of what this data contains."
  ))
}

summarize_data(mtcars)

Batch processing with rate limiting

library(purrr)

process_texts <- function(texts, system_prompt, delay = 0.5) {
  map_chr(texts, \(text) {
    chat <- chat_claude(system_prompt = system_prompt)
    Sys.sleep(delay)
    chat$chat(text)
  })
}

reviews <- c(
  "Great product, highly recommend!",
  "Terrible, waste of money",
  "It's okay, nothing special"
)

sentiments <- process_texts(
  reviews,
  system_prompt = "Classify as positive/negative/neutral. One word."
)

Error Handling

safe_chat <- function(prompt, provider = "claude") {
  chat_fn <- switch(provider,
    "claude" = chat_claude,
    "openai" = chat_openai,
    "ollama" = chat_ollama
  )

  tryCatch({
    chat <- chat_fn()
    chat$chat(prompt)
  }, error = function(e) {
    warning("LLM Error: ", e$message)
    NA_character_
  })
}

Common Mistakes

1. Creating new chat for each message

# Wrong - loses context
chat_claude()$chat("My name is Bob")
chat_claude()$chat("What's my name?")  # Doesn't know!

# Right - reuse chat object
chat <- chat_claude()
chat$chat("My name is Bob")
chat$chat("What's my name?")  # "Bob"

2. Forgetting to set API keys

# Check if keys are set
Sys.getenv("OPENAI_API_KEY")
Sys.getenv("ANTHROPIC_API_KEY")

# Set them
usethis::edit_r_environ()
# Add: OPENAI_API_KEY=your-key
# Add: ANTHROPIC_API_KEY=your-key

3. Not handling rate limits

# Add delays in loops
results <- map(items, \(item) {
  Sys.sleep(0.5)  # Delay
  chat$chat(item)
})

Summary

Task	Code
OpenAI chat	`chat_openai()`
Claude chat	`chat_claude()`
Local Ollama	`chat_ollama()`
System prompt	`chat_*(system_prompt = "...")`
Send message	`chat$chat("message")`
Stream output	`chat$stream("message")`
Register tool	`chat$register_tool(...)`
Extract data	`chat$extract_data(text, type)`
View history	`chat$get_turns()`

ellmer provides consistent syntax across providers
Use system prompts to control response style
Reuse chat objects to maintain context
Use tool calling for R function integration
Use structured extraction for reliable data parsing

Sources

How to Use Gemini API in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

Gemini is Google’s most capable AI model. It excels at reasoning, coding, and multimodal tasks (text + images). The ellmer package provides a tidyverse-friendly way to use Gemini’s API in R.

Alternatives: See also OpenAI API, Claude API, or run models locally with Ollama for free.

What you’ll learn:

Set up Gemini API access
Use ellmer’s chat_gemini() function
Create multi-turn conversations
Use structured data extraction
Compare Gemini models

Getting Started

Install ellmer

install.packages("ellmer")
library(ellmer)

Get your API key

Go to Google AI Studio
Sign in with your Google account
Click “Get API key” in the left sidebar
Create a new API key

Note: Gemini API has a generous free tier for experimentation.

Set your API key

# Option 1: Set for current session
Sys.setenv(GOOGLE_API_KEY = "your-api-key-here")

# Option 2: Add to .Renviron (recommended)
usethis::edit_r_environ()
# Add: GOOGLE_API_KEY=your-api-key-here

Basic Chat

Create a chat session

library(ellmer)

chat <- chat_gemini()
chat$chat("What is R programming?")

Single question

chat <- chat_gemini()
response <- chat$chat("Explain what a data frame is in R.")
response

Specify model

# Use specific Gemini model
chat <- chat_gemini(model = "gemini-2.5-pro")

Available Models

Current Stable Models

Model	Best For	Context
`gemini-2.5-flash`	Best price-performance, reasoning	1M tokens
`gemini-2.5-pro`	Complex tasks, deep reasoning, coding	1M tokens
`gemini-2.5-flash-lite`	Fastest, most budget-friendly	1M tokens

Latest Preview Models

Model	Best For	Context
`gemini-3.1-pro-preview`	Advanced intelligence, agentic coding	1M tokens
`gemini-3-flash-preview`	Frontier performance at low cost	1M tokens
`gemini-3.1-flash-lite-preview`	Most economical newest model	1M tokens

# Gemini 2.5 Flash (recommended default)
chat <- chat_gemini(model = "gemini-2.5-flash")

# Gemini 2.5 Pro (most capable stable)
chat <- chat_gemini(model = "gemini-2.5-pro")

# Gemini 3.1 Pro Preview (cutting edge)
chat <- chat_gemini(model = "gemini-3.1-pro-preview")

Note: Gemini models support up to 1M token context windows, ideal for analyzing long documents. Use -preview models for the latest capabilities.

System Prompts

Control Gemini’s behavior:

chat <- chat_gemini(
  system_prompt = "You are an expert R programmer. Always provide working code examples. Be concise."
)

chat$chat("How do I calculate the mean of a column?")

Specialized assistants

# Data analysis assistant
analyst <- chat_gemini(
  system_prompt = "You are a data analyst expert in R and the tidyverse.
  When asked questions, provide clear explanations with code examples."
)

analyst$chat("How do I find outliers in my data?")

Multi-turn Conversations

ellmer maintains conversation history automatically:

chat <- chat_gemini()

# First message
chat$chat("I have a dataset of customer purchases.")

# Follow-up (Gemini remembers context)
chat$chat("How would I calculate total spending per customer?")

# Another follow-up
chat$chat("Now how do I visualize this?")

View conversation history

# See all turns
chat$get_turns()

Practical Examples

Generate R code

chat <- chat_gemini(
  system_prompt = "You are an R expert. Return only executable R code with comments. No explanations outside code."
)

code <- chat$chat("
Create a function that:
1. Takes a data frame and column name
2. Removes outliers (values beyond 1.5*IQR)
3. Returns the cleaned data frame
")

cat(code)

Explain existing code

chat <- chat_gemini()

code_to_explain <- "
mtcars |>
  group_by(cyl) |>
  summarise(across(where(is.numeric), mean)) |>
  pivot_longer(-cyl)
"

chat$chat(paste("Explain this R code step by step:", code_to_explain))

Debug errors

chat <- chat_gemini(
  system_prompt = "You are an R debugging expert. When shown errors, explain the cause and provide a fix."
)

error_message <- "Error in select(df, name) : object 'name' not found"

chat$chat(paste("I got this error:", error_message,
                "My code was: df |> select(name)"))

Structured Output

Extract data in a specific format. For more examples, see How to Extract Structured Data with LLMs.

Define a schema

review_schema <- type_object(
  sentiment = type_string("positive, negative, or neutral"),
  confidence = type_number("confidence score 0-1"),
  summary = type_string("one sentence summary")
)

Extract structured data

chat <- chat_gemini()

result <- chat$extract_data(
  "This product exceeded my expectations! Great quality and fast shipping.",
  type = review_schema
)

Access results

result$sentiment
# "positive"

result$confidence
# 0.95

Streaming Responses

For long responses, stream output:

chat <- chat_gemini()

# Stream response (prints as it generates)
chat$stream("Write a detailed guide to ggplot2 themes.")

Error Handling

safe_chat <- function(prompt) {
  tryCatch({
    chat <- chat_gemini()
    chat$chat(prompt)
  }, error = function(e) {
    message("API Error: ", e$message)
    NA
  })
}

result <- safe_chat("What is 2+2?")

Common errors

Error	Cause	Solution
401 Unauthorized	Invalid API key	Check GOOGLE_API_KEY
429 Rate limit	Too many requests	Add delays
400 Bad request	Invalid parameters	Check model name

Gemini vs Other Providers

Feature	Gemini	OpenAI	Claude
Context window	1M tokens	128k tokens	1M tokens
Free tier	Yes	No	No
Multimodal	Yes	Yes	Yes
Code quality	Good	Excellent	Excellent
Speed	Fast	Fast	Medium
Latest models	Gemini 3.1	GPT-5.4 / o3	Claude 4.6

Choose Gemini when:

You need to process very long documents (1M tokens)
You want a free tier for experimentation
You’re already in the Google ecosystem
You need the latest preview models

Batch Processing

Process multiple texts with rate limiting:

library(purrr)

classify_text <- function(text) {
  chat <- chat_gemini(
    system_prompt = "Classify as positive/negative/neutral. One word only."
  )
  Sys.sleep(0.5)  # Rate limiting
  chat$chat(text)
}

texts <- c("Great product!", "Terrible service", "It's okay I guess")

results <- map_chr(texts, classify_text)
# "positive", "negative", "neutral"

Common Mistakes

1. Using wrong API key variable

# Wrong
Sys.setenv(GEMINI_API_KEY = "key")

# Right
Sys.setenv(GOOGLE_API_KEY = "key")

2. Forgetting conversation state

# Each chat_gemini() creates a NEW conversation
chat1 <- chat_gemini()
chat1$chat("My name is Alice")

chat2 <- chat_gemini()  # New conversation!
chat2$chat("What's my name?")  # Gemini doesn't know

# Reuse same chat object for context
chat1$chat("What's my name?")  # "Alice"

3. Not handling rate limits

# Add delays in loops
results <- map(items, \(item) {
  Sys.sleep(0.5)  # Important!
  chat$chat(item)
})

Summary

Task	Code
Basic chat	`chat <- chat_gemini(); chat$chat("Hi")`
Set API key	`Sys.setenv(GOOGLE_API_KEY = "key")`
System prompt	`chat_gemini(system_prompt = "...")`
Stable model	`chat_gemini(model = "gemini-2.5-flash")`
Latest model	`chat_gemini(model = "gemini-3.1-pro-preview")`
Structured output	`chat$extract_data(text, type)`

Gemini 2.5 models are stable with 1M token context
Gemini 3.x preview models offer cutting-edge capabilities
Free tier available for experimentation
Use system prompts to control response style

Sources

How to Use OpenAI API in R

Fri, 03 Apr 2026 18:30:00 GMT

Introduction

OpenAI’s API gives you access to powerful language models like GPT-4 and GPT-4o directly from R. You can generate text, answer questions, analyze data, and build AI-powered applications.

Alternatives: If you prefer Claude or want to run models locally with Ollama, the ellmer package provides a unified interface for all providers.

What you’ll learn: - Set up OpenAI API access - Use the openai R package - Create chat completions - Handle responses and errors - Manage costs with token limits

Getting Started

Install the openai package

install.packages("openai")
library(openai)

Get your API key

Create an account at platform.openai.com
Go to API Keys section
Create a new secret key
Add billing information (API access requires payment)

Set your API key

# Option 1: Set for current session
Sys.setenv(OPENAI_API_KEY = "sk-your-key-here")

# Option 2: Add to .Renviron (recommended)
usethis::edit_r_environ()
# Add: OPENAI_API_KEY=sk-your-key-here

Create Chat Completions

The main function for interacting with ChatGPT:

library(openai)

response <- create_chat_completion(
  model = "gpt-5.4-mini",
  messages = list(
    list(role = "user", content = "What is R programming?")
  )
)

# Extract the response text
response$choices[[1]]$message$content

With system prompt

Control the assistant’s behavior:

response <- create_chat_completion(
  model = "gpt-5.4-mini",
  messages = list(
    list(role = "system", content = "You are a helpful R programming tutor."),
    list(role = "user", content = "Explain what a data frame is.")
  )
)

Multi-turn conversation

messages <- list(
  list(role = "system", content = "You are an R expert."),
  list(role = "user", content = "How do I read a CSV file?"),
  list(role = "assistant", content = "Use read.csv() or readr::read_csv()..."),
  list(role = "user", content = "What's the difference between them?")
)

response <- create_chat_completion(
  model = "gpt-5.4-mini",
  messages = messages
)

Available Models

GPT-5 Series (Latest)

Model	Best For	Cost
`gpt-5.4`	Complex reasoning, coding	Higher
`gpt-5.4-mini`	Fast, everyday tasks	Medium
`gpt-5.4-nano`	Lowest latency, budget	Lower

O-Series (Reasoning Models)

Model	Best For	Cost
`o3`	Most powerful reasoning	Highest
`o3-mini`	Balanced reasoning	Medium
`o4-mini`	Affordable reasoning	Lower

Legacy Models (still available in API)

Model	Notes
`gpt-4o`	Retired from ChatGPT Feb 2026, API still available
`gpt-4-turbo`	128k context

# List available models
models <- list_models()

Control Response Parameters

Temperature (creativity)

# More deterministic (good for code)
create_chat_completion(
  model = "gpt-5.4-mini",
  messages = list(list(role = "user", content = "Write R code to calculate mean")),
  temperature = 0.2
)

# More creative (good for brainstorming)
create_chat_completion(
  model = "gpt-5.4-mini",
  messages = list(list(role = "user", content = "Suggest project ideas")),
  temperature = 0.9
)

Limit response length

response <- create_chat_completion(
  model = "gpt-5.4-mini",
  messages = list(list(role = "user", content = "Explain ggplot2")),
  max_tokens = 150  # Limit response length
)

Practical Examples

Generate R code

Use a system prompt to get code-only responses:

prompt <- "Write R code to:
1. Load the mtcars dataset
2. Calculate average mpg by cylinder count
3. Create a bar chart of the results"

Set low temperature for deterministic code:

response <- create_chat_completion(
  model = "gpt-5.4",
  messages = list(
    list(role = "system", content = "Return only R code, no explanations."),
    list(role = "user", content = prompt)
  ),
  temperature = 0.2
)

cat(response$choices[[1]]$message$content)

Analyze text data

Create a reusable sentiment classifier. For more robust extraction, see How to Extract Structured Data with LLMs.

analyze_sentiment <- function(text) {
  response <- create_chat_completion(
    model = "gpt-5.4-mini",
    messages = list(
      list(role = "system", content = "Classify as positive/negative/neutral. One word only."),
      list(role = "user", content = text)
    ),
    temperature = 0
  )
  response$choices[[1]]$message$content
}

Apply to multiple reviews:

reviews <- c("Great product!", "Terrible quality", "It's okay")
sapply(reviews, analyze_sentiment)
# "positive", "negative", "neutral"

Summarize text

long_text <- "Your long document here..."

response <- create_chat_completion(
  model = "gpt-5.4-mini",
  messages = list(
    list(role = "system", content = "Summarize the following text in 2-3 sentences."),
    list(role = "user", content = long_text)
  )
)

Using httr2 Directly

For more control, call the API directly with httr2.

Build the request

library(httr2)

response <- request("https://api.openai.com/v1/chat/completions") |>
  req_headers(
    Authorization = paste("Bearer", Sys.getenv("OPENAI_API_KEY")),
    `Content-Type` = "application/json"
  ) |>
  req_body_json(list(
    model = "gpt-5.4-mini",
    messages = list(list(role = "user", content = "Hello!"))
  )) |>
  req_perform() |>
  resp_body_json()

Extract the response

response$choices[[1]]$message$content

Error Handling

safe_completion <- function(prompt) {
  tryCatch({
    response <- create_chat_completion(
      model = "gpt-5.4-mini",
      messages = list(list(role = "user", content = prompt))
    )
    response$choices[[1]]$message$content
  }, error = function(e) {
    message("API Error: ", e$message)
    NA
  })
}

# Use safely
result <- safe_completion("What is 2+2?")

Common errors

Error	Cause	Solution
401 Unauthorized	Invalid API key	Check OPENAI_API_KEY
429 Rate limit	Too many requests	Add delays between calls
400 Bad request	Invalid parameters	Check model name, messages format

Cost Management

Understand tokens

# Rough estimate: 1 token ≈ 4 characters in English
# "Hello, how are you?" ≈ 6 tokens

# Check usage in response
response <- create_chat_completion(
  model = "gpt-5.4-mini",
  messages = list(list(role = "user", content = "Hi"))
)

response$usage
# $prompt_tokens: 9
# $completion_tokens: 10
# $total_tokens: 19

Limit costs

# Use cheaper models for simple tasks
create_chat_completion(
  model = "gpt-5.4-mini",  # Cheaper than gpt-4o
  messages = list(list(role = "user", content = prompt)),
  max_tokens = 100  # Limit response length
)

Batch Processing

Process multiple items efficiently:

library(purrr)

texts <- c("Text 1", "Text 2", "Text 3")

# Add delay to avoid rate limits
results <- map(texts, \(text) {
  Sys.sleep(0.5)  # 500ms delay

  response <- create_chat_completion(
    model = "gpt-5.4-mini",
    messages = list(
      list(role = "user", content = paste("Summarize:", text))
    )
  )
  response$choices[[1]]$message$content
})

Alternative: ellmer Package

The tidyverse-friendly alternative:

install.packages("ellmer")
library(ellmer)

chat <- chat_openai(model = "gpt-5.4-mini")
chat$chat("What is R programming?")

See our ellmer tutorial for more details.

Common Mistakes

1. Forgetting to set API key

# Check if key is set
Sys.getenv("OPENAI_API_KEY")
# Should not be empty

2. Not handling rate limits

# Add delays in loops
for (item in items) {
  result <- create_chat_completion(...)
  Sys.sleep(1)  # Wait 1 second
}

3. Using expensive models for simple tasks

# gpt-5.4-mini is often sufficient and much cheaper
# Only use gpt-5.4 or o3 for complex reasoning tasks

4. Not checking response structure

# Always verify the response has expected content
if (!is.null(response$choices[[1]]$message$content)) {
  result <- response$choices[[1]]$message$content
}

Summary

Task	Code
Basic completion	`create_chat_completion(model, messages)`
Set API key	`Sys.setenv(OPENAI_API_KEY = "key")`
Control creativity	`temperature = 0.2` (low) to `0.9` (high)
Limit response	`max_tokens = 100`
Check usage	`response$usage$total_tokens`

Start with gpt-5.4-mini for cost efficiency
Use o3 or o3-mini for complex reasoning tasks
Use low temperature for deterministic outputs
Add delays when processing multiple items
Always handle errors gracefully

Sources

How to create all combinations with expand_grid() in R

Thu, 02 Apr 2026 18:30:00 GMT

Introduction

Creating combinations of data is a common task in data analysis, especially when you need to explore all possible pairings between variables or expand datasets for modeling. R provides powerful functions like expand_grid() and expand.grid() that generate all combinations of input values, making it easy to create comprehensive datasets for analysis and visualization.

Loading Required Packages

First, let’s load the tidyverse package which contains the expand_grid() function we’ll be using throughout this tutorial.

library(tidyverse)

The tidyverse provides modern data manipulation tools that work seamlessly together for data analysis tasks.

Basic Combinations with expand_grid()

Let’s start by creating simple vectors and then generating all possible combinations between them.

var1 <- letters[1:5]
var1

This creates a vector with the first 5 lowercase letters. The letters constant in R contains all 26 lowercase letters of the alphabet.

var2 <- LETTERS[1:5]
var2

Similarly, this creates a vector with the first 5 uppercase letters using R’s built-in LETTERS constant.

Now let’s create all possible combinations of these two vectors:

combination_df <- expand_grid(var1 = letters[1:5],
                             var2 = LETTERS[1:5])
combination_df

The expand_grid() function creates a tibble with 25 rows (5 × 5), showing every possible pairing between the lowercase and uppercase letters.

Comparing expand_grid() vs expand.grid()

R also has a base function called expand.grid() that performs similar operations but with different output formatting.

combination_df_base <- expand.grid(var1 = letters[1:5],
                                  var2 = LETTERS[1:5])
combination_df_base

While expand.grid() produces similar results, expand_grid() returns a tibble and generally integrates better with tidyverse workflows.

Working with Data Frames

You can also use expand_grid() to expand existing data frames with additional variables. Let’s create a simple data frame first:

df <- tibble(year = 2021, 
            quarter = paste0("Q", 1:4))
df

This creates a tibble with one year and four quarters. The paste0() function concatenates “Q” with numbers 1-4.

Now let’s expand this data frame to include multiple companies:

expanded_df <- expand_grid(df, 
                          companies = c("GOOG", "MSFT", "NVDA"))
expanded_df

This operation creates 12 rows (4 quarters × 3 companies), duplicating the year and quarter information for each company. This is particularly useful when preparing data for financial analysis or creating templates for data collection.

Practical Applications

The combination functions are especially valuable for: - Creating factorial experimental designs - Generating parameter grids for model tuning - Building comprehensive datasets for simulation studies - Preparing data templates for reporting across multiple categories

Summary

The expand_grid() function provides an efficient way to create all possible combinations of variables in R. Whether you’re working with simple vectors or complex data frames, this function helps you generate comprehensive datasets needed for analysis, modeling, and reporting. Remember to use expand_grid() for tidyverse workflows and expand.grid() when working primarily with base R functions.

Effect of centering and scaling data in R

Thu, 02 Apr 2026 18:30:00 GMT

Introduction

Data scaling and centering are essential preprocessing steps in data analysis and machine learning. These transformations help normalize variables with different units and scales, making them comparable and improving the performance of many algorithms. We’ll explore how to visualize and apply these transformations using R and the Palmer penguins dataset.

Loading Required Libraries

Let’s start by loading the necessary packages for our analysis.

library(tidyverse)
library(palmerpenguins)
library(ggridges)
theme_set(theme_bw(16))

Preparing the Data

First, we’ll prepare our dataset by removing missing values and selecting only the numeric variables.

df <- penguins |>
  drop_na() |>
  select(-year) |>
  select(where(is.numeric))

Let’s examine the first few rows to understand our data structure.

df |> head()

Our dataset now contains four numeric variables: bill length, bill depth, flipper length, and body mass, all measured on different scales.

Visualizing Raw Data Distributions

Before applying any transformations, let’s visualize how our variables are distributed using boxplots.

df |>
  mutate(row_id = row_number()) |>
  pivot_longer(-row_id, names_to = "feature", values_to = "value") |>
  ggplot(aes(x = feature, y = value, color = feature)) +
  geom_boxplot(outlier.shape = NA) +
  geom_jitter(width = 0.1) +
  labs(title = "Raw Penguin Measurements on Different Scales",
       x = "Feature", y = "Value") +
  theme(legend.position = "none")

Boxplot with jittered points of raw penguin bill length, bill depth, flipper length, and body mass in R showing how the body mass variable dominates the shared y-axis because its numeric range is orders of magnitude larger than the other features before centering and scaling.

Notice how the variables have very different ranges - body mass is in thousands while bill measurements are in tens. This makes direct comparison difficult.

Using Ridge Plots for Better Comparison

Ridge plots provide a clearer view of each variable’s distribution shape.

df |>
  mutate(row_id = row_number()) |>
  pivot_longer(-row_id, names_to = "feature", values_to = "value") |>
  ggplot(aes(y = feature, x = value, fill = feature)) +
  geom_density_ridges2() +
  theme(legend.position = "none")

The different scales make it challenging to compare distribution shapes across variables.

Scaling and Centering Data

Now let’s apply both centering (subtracting the mean) and scaling (dividing by standard deviation) to standardize our variables.

df_scaled <- df |>
  scale(center = TRUE, scale = TRUE)

Let’s examine the transformed data.

df_scaled |> head()

The scaled data now has a mean of 0 and standard deviation of 1 for each variable, making them directly comparable.

Visualizing Scaled Data

Let’s see how the scaling transformation affects our distributions.

df_scaled |>
  as.data.frame() |>
  mutate(row_id = row_number()) |>
  pivot_longer(-row_id, names_to = "feature", values_to = "value") |>
  ggplot(aes(y = feature, x = value, fill = feature)) +
  geom_density_ridges2() +
  labs(title = "After Centering and Scaling (mean 0, sd 1)",
       x = "Standardized value", y = "Feature") +
  theme(legend.position = "none")

Ridge density plot of penguin bill length, bill depth, flipper length, and body mass after centering and scaling in R with scale(center = TRUE, scale = TRUE), showing all four distributions aligned on a common z-score axis with mean zero and unit standard deviation.

Now all variables are on the same scale, making it easy to compare their distribution shapes and identify which variables have the most variability.

Centering Only (Without Scaling)

Sometimes you might want to center data without scaling. This shifts distributions to have a mean of 0 but preserves the original variance.

df_centered <- df |>
  scale(center = TRUE, scale = FALSE)

Let’s examine the centered-only data.

df_centered |> head()

Visualizing Centered Data

Here’s how centering without scaling affects our distributions.

df_centered |>
  as.data.frame() |>
  mutate(row_id = row_number()) |>
  pivot_longer(-row_id, names_to = "feature", values_to = "value") |>
  ggplot(aes(y = feature, x = value, fill = feature)) +
  geom_density_ridges2() +
  labs(title = "After Centering Only (mean 0, original spreads)",
       x = "Centered value", y = "Feature") +
  theme(legend.position = "none")

Ridge density plot of centered-only penguin measurements in R using scale(center = TRUE, scale = FALSE), showing body mass retaining its much wider variance compared to the bill and flipper measurements while every distribution is now centered around zero.

The distributions maintain their original spreads but are now centered around zero, which can be useful for certain analyses while preserving the relative scale differences.

Summary

Data scaling and centering are powerful preprocessing techniques that make variables comparable and improve analysis quality. Use full scaling (center = TRUE, scale = TRUE) when you want all variables on the same scale, such as for machine learning algorithms. Use centering only (center = TRUE, scale = FALSE) when you want to preserve relative scale differences but center distributions around zero. Visual exploration with ridge plots helps you understand the impact of these transformations on your data.

How to calculate covariance in R

Thu, 02 Apr 2026 18:30:00 GMT

Introduction

Covariance and correlation matrices are fundamental tools in data analysis that help us understand relationships between multiple variables. Covariance measures how variables change together, while correlation standardizes these relationships to a -1 to 1 scale. This tutorial demonstrates how to calculate and interpret these matrices using R’s built-in functions and matrix operations.

Loading Required Packages

First, let’s load the necessary packages for our analysis:

library(tidyverse)
library(palmerpenguins)

Preparing the Data

We’ll use the Palmer penguins dataset, removing missing values and the year column to focus on the main measurements:

penguins <- penguins |>
  drop_na() |>
  select(-year)

Let’s examine our cleaned dataset to see what variables we’re working with. This gives us a complete dataset with species, island, sex, and four numeric measurements for each penguin.

Basic Covariance Matrix

To calculate covariance between all numeric variables, we first select only the numeric columns:

numeric_vars <- penguins |>
  select(is.numeric)

Now we can compute the covariance matrix using the cov() function:

cov(numeric_vars)

The covariance matrix shows how each pair of variables varies together. Larger absolute values indicate stronger relationships, but the scale depends on the units of measurement.

Standardizing the Data

To better compare relationships, let’s standardize our numeric variables to have mean 0 and standard deviation 1:

scaled_data <- numeric_vars |>
  scale()

Scaling puts all variables on the same scale, making covariances more comparable across different measurements.

Covariance of Scaled Data

With scaled data, the covariance matrix becomes more interpretable:

cov(scaled_data)

For standardized variables, covariance values range from -1 to 1, making them equivalent to correlation coefficients.

Correlation Matrix

We can also calculate correlations directly using the cor() function:

cor(scaled_data)

The correlation matrix is identical to the covariance matrix of scaled data. Values closer to 1 or -1 indicate stronger linear relationships.

Calculating Covariance Between Variable Groups

Sometimes we want to examine relationships between specific groups of variables. Here we compare bill measurements with body measurements:

bill_vars <- scaled_data[, 1:2]  # bill_length_mm, bill_depth_mm
body_vars <- scaled_data[, 3:4]  # flipper_length_mm, body_mass_g

cor(bill_vars, body_vars)

This cross-correlation matrix shows how bill dimensions relate to body size measurements.

Manual Covariance Calculation

Understanding the mathematical foundation, we can calculate covariance manually using matrix multiplication:

n <- nrow(scaled_data)
manual_cov <- (t(scaled_data) %*% scaled_data) / (n - 1)

This matrix multiplication approach gives us the same result as cov(). The formula represents the mathematical definition: the average of cross-products of deviations from means.

Summary

Covariance and correlation matrices provide powerful insights into variable relationships in multivariate data. While cov() and cor() functions handle the calculations efficiently, understanding the underlying matrix operations helps deepen your statistical intuition. Standardizing variables before analysis often makes results more interpretable, especially when variables have different units or scales.

How to calculate Pearson correlation in R

Thu, 02 Apr 2026 18:30:00 GMT

Introduction

Data scaling and centering are fundamental preprocessing steps in data analysis and machine learning. Scaling transforms variables to have similar ranges, while centering adjusts variables to have a mean of zero. These transformations are essential when working with variables of different units or magnitudes, ensuring that no single variable dominates analyses due to its scale.

Setup and Data Preparation

Let’s start by loading the necessary libraries and preparing our dataset.

library(tidyverse)
library(palmerpenguins)
theme_set(theme_bw(16))

We’ll use the Palmer Penguins dataset, selecting only the numeric variables for our scaling demonstration.

df <- penguins |>
  drop_na() |>
  select(-year) |>
  select(where(is.numeric))

Let’s examine our cleaned dataset structure:

df |> head()

This gives us four numeric variables: bill length, bill depth, flipper length, and body mass, all measured in different units and scales.

Understanding Variable Scales

Before scaling, let’s visualize how our variables differ in their ranges and distributions.

df |>
  mutate(row_id = row_number()) |>
  pivot_longer(-row_id, names_to = "feature", values_to = "value") |>
  ggplot(aes(x = feature, y = value, color = feature)) +
  geom_boxplot(outlier.shape = NA) +
  geom_jitter(width = 0.1) +
  scale_y_log10() +
  theme(legend.position = "none")

Boxplot in R showing Palmer Penguins numeric features on a log scale before Pearson correlation calculation, highlighting the different scales of bill length, bill depth, flipper length, and body mass in ggplot2

Notice how body mass ranges in thousands while bill measurements are in tens - this dramatic difference in scales can problematic for many analyses.

Standardization (Z-score Scaling)

Standardization transforms variables to have a mean of 0 and standard deviation of 1. This is the default behavior of the scale() function.

df_scaled <- df |> scale()
df_scaled |> head()

Let’s visualize the scaled data to see how the distributions now compare:

df_scaled |>
  as.data.frame() |>
  mutate(row_id = row_number()) |>
  pivot_longer(-row_id, names_to = "feature", values_to = "value") |>
  ggplot(aes(x = feature, y = value, color = feature)) +
  geom_boxplot(outlier.shape = NA) +
  geom_jitter(width = 0.1) +
  theme(legend.position = "none")

Boxplot in R of standardized penguin features (mean 0, standard deviation 1) used to calculate Pearson correlation, showing comparable ranges after scale() transformation in ggplot2

Now all variables have similar scales, making them directly comparable. The standardized variables are centered around 0 with similar spreads.

Centering Only

Sometimes you may want to center variables (mean = 0) without scaling them. Use the center and scale parameters explicitly:

df_centered <- df |>
  scale(center = TRUE, scale = FALSE)
df_centered |> head()

Centering alone shifts the distribution but preserves the original scale relationships between variables.

Verifying Transformations

Let’s confirm our scaling worked by checking the means of our standardized data:

colMeans(df_scaled)

All means should be essentially zero (within floating-point precision). Now let’s check the covariance matrix:

cov(df_scaled)

For standardized data, the covariance matrix equals the correlation matrix, since standardization makes the variance of each variable equal to 1.

Working with Correlation Matrices

The correlation matrix shows the linear relationships between variables:

cor(df_scaled)

You can also calculate correlations between subsets of variables:

cor(df_scaled[,1:2], df_scaled[,3:4])

This shows correlations between the first two variables (bill measurements) and the last two (flipper length and body mass).

Matrix Multiplication Approach

Understanding that correlation can be computed through matrix multiplication helps grasp the mathematical foundation:

n <- nrow(df_scaled)
(t(df_scaled) %*% df_scaled) / (n - 1)

This manual calculation produces the same result as the cor() function, demonstrating how correlation matrices are computed mathematically.

Let’s verify this matches the built-in function:

cor(df_scaled)

Both approaches yield identical results, confirming our understanding of the underlying mathematics.

Summary

Data scaling and centering are crucial preprocessing steps that ensure variables contribute equally to analyses. Standardization (z-score scaling) creates variables with mean 0 and standard deviation 1, while centering only adjusts the mean. The scale() function in R provides flexible options for these transformations, and understanding the relationship between scaled data and correlation matrices helps build intuition for multivariate analyses. Always visualize your data before and after scaling to ensure the transformations achieve your analytical goals.

How to format currency in gt tables in R

Thu, 02 Apr 2026 18:30:00 GMT

Introduction

The gt package provides powerful tools for creating beautiful, publication-ready tables in R. One of its most useful features is fmt_currency(), which automatically formats numeric values as currency with proper symbols, decimal places, and locale-specific formatting. This function is essential when creating financial reports, dashboards, or any data presentation involving monetary values.

Loading Required Libraries

First, let’s load the necessary packages for this tutorial:

library(gt)
library(tidyverse)

Creating a Basic Data Table

Let’s start with a simple dataset containing stock symbols and their prices:

stock_data <- tibble(
  symbol = c("GOOG", "META", "MSFT"), 
  price = c(168, 465, 425)
)
stock_data

This creates a basic tibble with three tech stocks and their current prices in dollars.

Converting to a gt Table

Now let’s convert our data frame into a gt table for better presentation:

stock_data |>
  gt()

The gt() function transforms our tibble into a formatted HTML table, but the prices still appear as plain numbers without currency formatting.

Basic Currency Formatting

To format all numeric columns as currency, we can use fmt_currency() without specifying columns:

stock_data |>
  gt() |>
  fmt_currency()

This applies dollar formatting to all numeric columns, adding the $ symbol and two decimal places.

Targeting Specific Columns

For more control, specify which columns to format using the columns parameter:

stock_data |>
  gt() |>
  fmt_currency(columns = price)

This approach is safer when you have multiple numeric columns but only want to format certain ones as currency.

Working with Multiple Currencies

Let’s create a more complex example with multiple currency columns:

stock_data_multi <- tibble(
  symbol = c("GOOG", "META", "MSFT"), 
  price_USD = c(168, 465, 425)
) |>
  mutate(price_INR = price_USD * 80)

stock_data_multi

This creates a dataset with prices in both US dollars and Indian rupees.

Exploring Available Currency Options

Before formatting different currencies, let’s see what currency symbols are available:

info_currencies(type = "symbol")

This function shows all supported currency symbols and their corresponding names in the gt package.

Formatting Multiple Currencies

Now let’s format each currency column appropriately:

stock_data_multi |>
  gt() |>
  fmt_currency(columns = price_USD) |>
  fmt_currency(columns = price_INR, currency = "rupee")

Each fmt_currency() call can target different columns and apply different currency symbols, making it easy to create multi-currency tables.

Dynamic Currency Formatting

For more advanced use cases, you can format currencies dynamically based on data in your table:

multi_locale_data <- tibble(
  amount = rep(50.84, 5),
  currency = c("JPY", "USD", "GHS", "KRW", "CNY"),
  locale = c("ja", "en", "ee", "ko", "zh")
)

multi_locale_data

This creates a dataset with the same amount in different currencies and their corresponding locales.

Using Locale-Based Formatting

Finally, let’s format the currencies using their appropriate locales and hide the helper columns:

multi_locale_data |>
  gt() |>
  fmt_currency(
    columns = amount,
    locale = from_column(column = "locale")
  ) |>
  cols_hide(columns = locale)

The from_column() function tells gt to use values from the locale column to determine formatting, while cols_hide() removes the helper column from the final display.

Summary

The fmt_currency() function in gt provides flexible currency formatting options, from basic dollar formatting to complex multi-locale presentations. Key features include targeting specific columns, using different currency symbols, and dynamic formatting based on data values. These tools make it easy to create professional financial reports and data presentations with properly formatted monetary values.

How to count unique values with n_distinct() in R

Wed, 25 Mar 2026 18:30:00 GMT

Introduction

The n_distinct() function in dplyr is a powerful tool for counting the number of unique values in a vector or dataset. This function is essential for data exploration and quality checks, helping you quickly understand the diversity of your data. You’ll find it particularly useful when examining categorical variables, checking for duplicates, or summarizing data by groups.

Setup

Let’s start by loading the tidyverse package and creating some sample data to work with:

library(tidyverse)

# Create sample data with some duplicate IDs and a missing value
df <- tibble(
  id = c(2, 4, 1, 2, 3, 4, NA),
  amount = c(250, 200, 150, 250, 300, 120, 200)
)

df

Our dataset contains 7 rows but notice that some ID values are repeated, and we have one missing value (NA).

Basic Usage of n_distinct()

The most common way to count distinct values is to use n_distinct() with pull() to extract a column:

df |>
  pull(id) |> 
  n_distinct()

This counts all unique values including NA, so we get 5 distinct values (1, 2, 3, 4, and NA).

Handling Missing Values

To exclude missing values from the count, use the na.rm parameter:

df |>
  pull(id) |> 
  n_distinct(na.rm = TRUE)

Now we get 4 distinct values, excluding the NA.

Alternative Approaches

You can achieve the same result using unique() and length():

df |>
  pull(id) |> 
  unique() |>
  length()

This approach first gets unique values, then counts them. However, n_distinct() is more concise and handles missing values more explicitly.

Direct Column Access

You can also use n_distinct() directly on a column without pipes:

n_distinct(df$id)

This base R syntax is shorter for simple cases but doesn’t integrate as well with dplyr workflows.

Counting Distinct Rows

When applied to an entire dataframe, n_distinct() counts unique combinations of all columns:

df |>
  n_distinct()

This tells us how many completely unique rows exist in our dataset.

Using n_distinct() with Group Operations

One of the most powerful applications is counting distinct values within groups:

# Example with grouped data
df_grouped <- tibble(
  category = c("A", "A", "B", "B", "B"),
  value = c(10, 20, 10, 30, 20)
)

df_grouped |>
  group_by(category) |>
  summarize(distinct_values = n_distinct(value))

This shows how many distinct values exist within each category, which is invaluable for understanding data distribution across groups.

Summary

The n_distinct() function is an essential tool for exploratory data analysis in R. Use it to quickly count unique values in columns, check data quality, and summarize categorical variables. Remember to use na.rm = TRUE when you want to exclude missing values from your counts. Combined with group_by(), it becomes even more powerful for understanding patterns within different subsets of your data.

How to get the first row of each group in R

Wed, 25 Mar 2026 18:30:00 GMT

Introduction

Calculating proportions and frequencies is a fundamental task in data analysis that helps us understand the relative distribution of categorical variables. This tutorial demonstrates how to use dplyr functions like count() and mutate() to calculate proportions from grouped data. These techniques are essential when you need to convert raw counts into percentages or when comparing the relative sizes of different groups in your dataset.

Setup

First, let’s load the required packages and prepare our data for analysis.

library(tidyverse)
library(palmerpenguins)

We’ll use the Palmer penguins dataset, but first we need to remove any missing values to ensure accurate calculations.

penguins_clean <- penguins |>
  drop_na()

This gives us a clean dataset with complete observations for all variables.

Basic Counting

Let’s start by counting the number of penguins by species to understand our data structure.

penguins_clean |>
  count(species)

The count() function returns the frequency of each species in our dataset. This shows us the absolute numbers, but often we want to know the relative proportions.

Calculating Simple Proportions

To convert counts into proportions, we can use mutate() to create a new column that divides each count by the total.

penguins_clean |>
  count(species) |>
  mutate(prop = n / sum(n))

This creates a proportion column where all values sum to 1.0, showing us what fraction of the total each species represents.

Alternative Approach with prop.table()

R provides the prop.table() function as an alternative way to calculate proportions from counts.

penguins_clean |>
  count(species) |>
  mutate(freq = prop.table(n))

Both approaches yield the same results, but prop.table() can be more explicit about your intention to calculate proportions.

Proportions with Multiple Groups

When working with multiple categorical variables, calculating proportions becomes more complex. Let’s count penguins by both species and sex.

penguins_clean |>
  count(species, sex)

This gives us the count for each combination of species and sex.

Overall Proportions Across Groups

To calculate what proportion each species-sex combination represents of the total dataset:

penguins_clean |>
  count(species, sex) |>
  mutate(prop = n / sum(n))

Each proportion represents that group’s share of the entire dataset, and all proportions will sum to 1.0.

Proportions Within Groups

Often you’ll want to calculate proportions within each species rather than across the entire dataset.

penguins_clean |>
  count(species, sex) |>
  mutate(prop_within_species = n / sum(n), .by = species)

The .by argument tells mutate() to calculate proportions separately for each species. Now the proportions within each species sum to 1.0, showing the sex distribution within each species.

Summary

This tutorial covered several approaches for calculating proportions from categorical data using dplyr. The key techniques include using count() to get frequencies, mutate() with division to calculate simple proportions, and the .by argument to calculate proportions within specific groups. These methods are essential for exploratory data analysis and creating meaningful summaries of categorical variables in your datasets.

How to get the last row of each group in R

Wed, 25 Mar 2026 18:30:00 GMT

Introduction

The dplyr package provides powerful functions for extracting specific rows and performing grouped operations on data frames. These functions help you access the first, last, or nth observations in your data, calculate proportions within groups, and handle consecutive runs of data. These techniques are essential for data exploration, summary statistics, and data cleaning tasks.

Setup

Let’s start by loading the necessary packages and preparing our data:

library(tidyverse)
library(palmerpenguins)

We’ll clean the penguins dataset by removing any rows with missing values:

penguins <- penguins |>
  drop_na()

This gives us a complete dataset with 333 penguin observations to work with.

Extracting Specific Rows

Getting the Last Row

The last() function extracts the final row from your dataset:

penguins |>
  last()

This returns the complete last observation, showing all variables for the final penguin in the dataset.

Getting the First Row

Similarly, first() extracts the initial row:

penguins |>
  first()

You’ll see the first penguin observation with all its measurements and characteristics.

Getting the Nth Row

Use nth() to extract any specific row by position:

penguins |>
  nth(n = 10)

This retrieves the 10th row in the dataset, useful when you need a specific observation by its position.

Using Position Functions in Summaries

Summarizing with Last Values

You can use position functions within summarize() to extract specific values:

penguins |>
  summarize(last_species = last(species))

This creates a summary showing only the species of the last penguin in the dataset.

Summarizing with First Values

The same approach works for first values:

penguins |>
  summarize(first_species = first(species))

This tells you which species appears first in your ordered dataset.

Slicing Rows by Groups

Getting First Row per Group

Combine group_by() with slice_head() to get the first observation from each group:

penguins |>
  group_by(species) |>
  slice_head(n = 1)

This returns one penguin from each species - specifically the first one that appears for each species in the data.

Getting Single First Row

Without grouping, slice_head() returns the first row(s) from the entire dataset:

penguins |>
  slice_head(n = 1)

This gives you just the very first penguin observation.

Getting Last Row per Group

Use slice_tail() with grouping to get the final observation from each group:

penguins |>
  group_by(species) |>
  slice_tail(n = 1)

This returns the last penguin observation for each of the three species.

Calculating Proportions

Basic Proportions

Calculate what proportion each group represents of the total:

penguins |>
  count(species) |>
  mutate(prop = n / sum(n))

This shows both the count and proportion of each penguin species in the dataset.

Alternative Proportion Calculation

You can also use prop.table() for the same result:

penguins |>
  count(species) |>
  mutate(freq = prop.table(n))

Both approaches give you the relative frequency of each species as decimals that sum to 1.

Proportions with Multiple Variables

Calculate proportions across combinations of variables:

penguins |>
  count(species, sex) |>
  mutate(prop = n / sum(n))

This shows what proportion each species-sex combination represents of all penguins.

Working with Consecutive Values

Identifying Consecutive Groups

The consecutive_id() function helps identify runs of consecutive identical values:

df <- data.frame(x = c(0, 0, 1, 0), y = c(2, 2, 2, 2))
df

Let’s create groups based on consecutive runs of the same x and y values:

df |>
  group_by(id = consecutive_id(x, y), x, y)

This assigns a unique ID to each consecutive run of identical x,y combinations.

Summarizing Consecutive Groups

Count how many observations are in each consecutive group:

df |>
  group_by(id = consecutive_id(x, y), x, y) |>
  summarise(n = n())

This reveals that we have three distinct consecutive groups: two single observations and one pair of consecutive identical values.

Summary

These dplyr functions provide flexible ways to extract specific rows, work with grouped data, and calculate proportions. The first(), last(), and nth() functions are perfect for accessing specific observations, while slice_head() and slice_tail() work excellently with grouped data. Proportion calculations help you understand the relative frequency of different categories, and consecutive_id() is invaluable for identifying patterns in sequential data. Master these functions to make your data exploration and summarization tasks more efficient and insightful.

How to get top and bottom rows of each group in R

Wed, 25 Mar 2026 18:30:00 GMT

Introduction

The slice_max() function in dplyr is a powerful tool for selecting the top n rows with the highest values from your data. Unlike simple sorting, slice_max() lets you efficiently extract just the records you need, making it perfect for finding top performers, highest scores, or maximum values within groups. This function is especially useful when working with grouped data where you want to find the top entries for each category.

Getting Started

First, let’s load the tidyverse package and create some sample data to work with:

library(tidyverse)

We’ll create a dataset with symbols and values to demonstrate different uses of slice_max():

set.seed(2024)
df <- tibble(
  symbol = sample(letters, 10),
  value = rnorm(10, mean = 5, sd = 10)
)
df

This gives us a dataset with 10 random symbols and their corresponding numeric values, including both positive and negative numbers.

Basic Sorting vs slice_max()

Let’s first see what our data looks like when sorted by value:

df |> arrange(value)

The arrange() function sorts all rows, but what if we only want the top 3 highest values? This is where slice_max() becomes useful.

Finding Top Values

To get the 3 rows with the highest values, we use slice_max():

df |> slice_max(value, n = 3)

This returns only the top 3 rows with the highest values, which is much more efficient than sorting the entire dataset when you only need the top entries.

Working with Grouped Data

slice_max() becomes even more powerful when combined with group_by(). Let’s first create a grouping variable:

df_grouped <- df |>
  mutate(direction = ifelse(value > 0, "positive", "negative"))
df_grouped

Now we can find the top values within each group:

df_grouped |>
  group_by(direction) |>
  slice_max(value, n = 2)

This gives us the top 2 highest values for both positive and negative numbers separately.

Handling Ties

When there are tied values, slice_max() includes all tied observations by default. You can control this behavior:

# Include all ties (default)
df |> slice_max(value, n = 3)

# Keep exactly n rows, breaking ties randomly
df |> slice_max(value, n = 3, with_ties = FALSE)

The with_ties parameter determines whether to include all rows that tie for the nth position.

Practical Example with Real Data

Let’s use the penguins data to see a more realistic example:

library(palmerpenguins)

penguins |>
  filter(!is.na(body_mass_g)) |>
  group_by(species) |>
  slice_max(body_mass_g, n = 2)

This finds the 2 heaviest penguins of each species, which is useful for understanding the size distribution across different penguin types.

Summary

slice_max() is an efficient way to extract the top n rows based on a specific variable, especially when you don’t need to sort your entire dataset. It works particularly well with grouped data, allowing you to find top values within each category. Remember to handle missing values appropriately and consider the with_ties parameter when exact row counts matter. This function is a great alternative to combining arrange() and head(), providing cleaner and more intuitive code for common data analysis tasks.

How to update rows in a dataframe with rows_update() in R

Wed, 25 Mar 2026 18:30:00 GMT

Introduction

The rows_update() function in dplyr provides a powerful way to update existing rows in a data frame using values from another data frame. Similar to SQL’s UPDATE statement, it modifies rows where key columns match between two tables, making it ideal for applying corrections, updates, or changes to specific records without altering the overall structure of your data.

Setting Up

Let’s start by loading the required packages and checking our dplyr version:

library(tidyverse)
packageVersion("dplyr")

Basic Syntax and Parameters

The rows_update() function follows this general syntax:

rows_update(
  x,                    # target data frame to update
  y,                    # data frame with new values
  by = NULL,            # key columns for matching
  unmatched = "error",  # how to handle unmatched keys
  copy = FALSE,
  in_place = FALSE
)

The function requires that key values in the update data frame (y) are unique and, by default, must exist in the target data frame (x).

Basic Example: Updating Student Scores

Let’s create a practical example with student data where we need to update some test scores:

# Original student data
students <- tibble(
  student_id = 1:5,
  name = c("Alice", "Bob", "Charlie", "Liz", "Sam"),
  score = c(85, 90, 88, 92, 89)
)

Now let’s create a data frame with updated scores for specific students:

# Updated scores for students 1 and 5
score_updates <- tibble(
  student_id = c(1, 5),
  score = c(100, 98)
)

Let’s examine our original data first:

students

And our update data:

score_updates

Performing the Update

Now we can update the original data frame with the new scores:

updated_students <- students |>
  rows_update(score_updates)

Since both data frames share the student_id column, rows_update() automatically uses it as the key. Notice that only Alice’s and Sam’s scores changed to 100 and 98 respectively, while other rows remained unchanged.

Specifying Key Columns Explicitly

You can explicitly specify which column(s) to use for matching using the by parameter:

students |>
  rows_update(score_updates, by = "student_id")

This produces the same result but makes the key column explicit, which is helpful when working with complex data or when column names might be ambiguous.

Handling Extra Columns

What happens when the update data frame contains additional columns not present in the original data? Let’s see:

# Update data with an extra column
updates_with_grade <- tibble(
  student_id = c(1, 5),
  score = c(100, 98),
  grade = c("A", "A")
)

If we try to update with this data frame containing the extra grade column:

# This will cause an error
students |>
  rows_update(updates_with_grade, by = "student_id")

This will produce an error because rows_update() only updates existing columns and doesn’t add new ones. The original data frame structure is preserved.

Key Points and Best Practices

When using rows_update(), remember these important points:

Key values in the update data frame must be unique
Only existing columns in the target data frame will be updated
The function preserves the original structure and row order
Unmatched keys in the update data frame cause an error by default

Summary

The rows_update() function is an essential tool for maintaining and updating data frames in a controlled manner. It provides a safe way to apply updates to specific rows while preserving data integrity and structure. Use it when you need to modify existing records without adding new rows or columns, making it perfect for data correction workflows and incremental updates.

How to use anti_join() to find non-matching rows in R

Wed, 25 Mar 2026 18:30:00 GMT

Introduction

The anti_join() function in R’s dplyr package helps you find rows that exist in one dataset but not in another. This filtering join is particularly useful for identifying missing records, finding unique observations, or cleaning data by removing unwanted matches. Use anti_join() when you want to exclude rows based on matching keys between two datasets.

Setting Up the Data

Let’s start by loading the tidyverse package and creating two sample datasets to demonstrate anti_join().

library(tidyverse)

We’ll create our first dataset containing R packages with their corresponding IDs:

set.seed(123)
id <- sample(1:5, 3, replace = FALSE)
df1 <- tibble(
  id = id,
  packages = c("dplyr", "tidyr", "tibble")
)
df1

This creates a tibble with three R packages, each assigned a random ID. The set.seed() ensures reproducible results.

Now let’s create a second dataset with some overlapping packages but different IDs:

set.seed(123)
id <- sample(1:6, 3, replace = FALSE)
df2 <- tibble(
  id = id,
  packages = c("dplyr", "ggplot2", "tidyr")
)
df2

Notice that df2 contains “dplyr” and “tidyr” (also in df1) plus “ggplot2” (unique to df2).

Finding Rows in df1 but Not in df2

To find packages that exist in df1 but not in df2, we use anti_join() with df1 as the first argument:

anti_join(df1, df2, by = "packages")

This returns rows from df1 where the package name doesn’t have a match in df2. Since “dplyr” and “tidyr” appear in both datasets, only “tibble” (unique to df1) will be returned.

Finding Rows in df2 but Not in df1

We can reverse the operation to find packages in df2 that aren’t in df1:

anti_join(df2, df1, by = "packages")

This returns “ggplot2” since it’s the only package in df2 that doesn’t appear in df1.

Understanding the Join Key

When the column names are the same (like “packages” in both datasets), anti_join() automatically uses them as the join key:

# These are equivalent
anti_join(df1, df2, by = "packages")
anti_join(df1, df2)  # automatic matching

The function automatically detects matching column names and uses them for the join operation.

Summary

The anti_join() function is a powerful tool for identifying non-matching records between datasets. Remember that the order matters: anti_join(A, B) returns rows from A that don’t match B, while anti_join(B, A) returns rows from B that don’t match A. This makes it invaluable for data cleaning, validation, and finding unique observations across related datasets.

Rstats 101

How to Use Embeddings and Semantic Search in R

Introduction

How Embeddings Work

The Famous King–Queen Example

Getting Started

Get a Single Embedding

Call it on one string

Cosine Similarity: Comparing Two Embeddings

Compare two sentences

Embed Many Documents at Once

Build an embedding store

Visualizing Semantic Similarity

Build a Similarity Matrix

Plot a Heatmap with ggplot

Plot a 2D Projection

Plot the projection

Visualize Query Distance

Anti-Correlation: When Sentences Disagree

A Mixed Set of Sentences

Plot the Mixed Heatmap

The Surprising Antonym Result

Visualize the Three Clusters

Build a Semantic Search Function

Run a query

Practical Example: Search FAQ Entries

Query the FAQ store

From Search to RAG: Answer Questions Over Your Docs

The RAG Function

Run a RAG Query

Why Retrieval Beats a Huge Prompt

Going Further with RAG

Caching Embeddings

Choosing a Model

Use Local Embeddings with Ollama

Performance Tips

Common Mistakes

Summary

Frequently Asked Questions

How much do embeddings cost in R?

Do I need a vector database to use embeddings in R?

Can I use open-source embedding models in R?

How do I update embeddings when my documents change?

How many dimensions should my embeddings have?

Are embeddings enough for sentiment analysis?

What’s the difference between embeddings and fine-tuning?

Can I build a chatbot with embeddings in R?

Related Posts

Sources

How to Analyze Sentiment with LLMs in R

Introduction

Why Use LLMs for Sentiment Analysis?

Getting Started

Basic Sentiment Analysis

Simple three-way classification

Why LLMs handle nuance better

Structured Sentiment Output

Using extract_data for reliable results

Extract sentiment

Sentiment with Confidence Scores

When you need more than just a label

Extract detailed sentiment

Fine-grained Sentiment

Five-point scale

Classify on the scale

Numeric scores

Aspect-Based Sentiment

Analyze sentiment by aspect

Extract aspect sentiments

Convert to data frame

Batch Sentiment Analysis

Process multiple reviews

Apply to a data frame

Calculate summary statistics

Domain-Specific Sentiment

Customize for your domain

Emotion Detection

Beyond positive/negative

Detect emotions

Error Handling