Test Drive: logprobs
Published on .
Today I’m test driving logprobs on OpenAI to understand the probability of output tokens. I’ll ask the model to classify a company’s industry given the description, then look at the logprobs to understand the confidence. There are other applications too long autocomplete and more.
The process invovles:
- Invoke the chat completions API with the
logprobsandtop_logprobsparams - Print the results
You should head over to OpenAI’s Cookbook on “Using logprobs” to go deeper than my test drive.
This is a very neat glance into the inner workings of the model which I did not know was possible!
Here’s the code.
from openai import OpenAI
import numpy as np
import textwrap
client = OpenAI()
description = """PTT Public Company Limited is a Thailand-based company engaged in the gas and petroleum businesses. The Company supplies, transports and distributes natural gas vehicle (NGV), petroleum products and lubricating oil via service stations throughout Thailand and also exports to overseas markets. Through its subsidiaries and affiliated companies, the Company is involved in exploration, production, refinery, marketing and distribution of petroleum, petrochemical products and aromatics. In addition, the Company operates international trade businesses, including import and export of crude oil, condensates, petroleum products, petrochemicals, and sourcing of international transport vessels and carriers."""
response = client.chat.completions.create(
messages=[{"role": "user", "content": textwrap.dedent(f"""\
You will be given a description of a company.
Classify the company into an industry
Return only the name of the industry, and nothing else.
Company Description: {description}""")}],
model="gpt-4o-mini",
logprobs=True,
top_logprobs=5,
)
print(response.choices[0].message.content)
for lp in response.choices[0].logprobs.content[0].top_logprobs:
print(lp.token, np.round(np.exp(lp.logprob)*100,3))Code language: Python (python)Here’s the output
Energy
Energy 56.193
Oil 43.763
Pet 0.04
Gas 0.003
O 0.0Code language: Python (python)