Test Drive: logprobs by Nat Taylor

Test Drive: logprobs

Published on Sep 24, 2024.

WordPress database error: [<div style="clear:both"> </div><div class="queries" style="clear:both; margin_bottom:2px; border: red dotted thin;">Queries made or created this session were<br/> <ol> <li>Raw query: SELECT * FROM wp_options WHERE </li> <li>Rewritten: SELECT * FROM wp_options WHERE </li> <li>With Placeholders: SELECT * FROM wp_options WHERE </li> <li>Prepare: SELECT * FROM wp_options WHERE </li> </ol> </div><div style="clear:both; margin_bottom:2px; border: red dotted thin;" class="error_message" style="border-bottom:dotted blue thin;">Error occurred at line 1644 in Function prepare_query. <br/> Error message was: Problem preparing the PDO SQL Statement. Error was: SQLSTATE[HY000]: General error: 1 incomplete input </div><pre>#0 /home/nattaylor/public_html/wordpress/wp-content/db.php(2746): WP_SQLite_DB\PDOEngine->get_error_message() #1 /home/nattaylor/public_html/wordpress/wp-content/db.php(3484): WP_SQLite_DB\wpsqlitedb->query('...') #2 /home/nattaylor/public_html/wordpress/wp-content/db.php(2952): WP_SQLite_DB\PDOSQLiteDriver->execute_duplicate_key_update() #3 /home/nattaylor/public_html/wordpress/wp-content/db.php(1893): WP_SQLite_DB\PDOSQLiteDriver->rewrite_query('...', '...') #4 /home/nattaylor/public_html/wordpress/wp-content/db.php(1357): WP_SQLite_DB\PDOEngine->execute_insert_query_new('...') #5 /home/nattaylor/public_html/wordpress/wp-content/db.php(2739): WP_SQLite_DB\PDOEngine->query('...') #6 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1143): WP_SQLite_DB\wpsqlitedb->query('...') #7 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1552): add_option('...', 1758231049, '', '...') #8 /home/nattaylor/public_html/wordpress/wp-content/plugins/syntax-highlighting-code-block/inc/functions.php(671): set_transient('...', Array, 2592000) #9 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-block.php(586): Syntax_Highlighting_Code_Block\render_block(Array, '...', Object(WP_Block)) #10 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2359): WP_Block->render() #11 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2431): render_block(Array) #12 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-hook.php(324): do_blocks('...') #13 /home/nattaylor/public_html/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters('...', Array) #14 /home/nattaylor/public_html/wordpress/wp-includes/post-template.php(256): apply_filters('...', '...') #15 /home/nattaylor/public_html/wordpress/wp-content/themes/ntdc/index.php(70): the_content() #16 /home/nattaylor/public_html/wordpress/wp-includes/template-loader.php(106): include('...') #17 /home/nattaylor/public_html/wordpress/wp-blog-header.php(19): require_once('...') #18 /home/nattaylor/public_html/wordpress/index.php(17): require('...') </pre>]
SELECT * FROM wp_options WHERE

WordPress database error: [<div style="clear:both"> </div><div class="queries" style="clear:both; margin_bottom:2px; border: red dotted thin;">Queries made or created this session were<br/> <ol> <li>Raw query: SELECT * FROM wp_options WHERE </li> <li>Rewritten: SELECT * FROM wp_options WHERE </li> <li>With Placeholders: SELECT * FROM wp_options WHERE </li> <li>Prepare: SELECT * FROM wp_options WHERE </li> </ol> </div><div style="clear:both; margin_bottom:2px; border: red dotted thin;" class="error_message" style="border-bottom:dotted blue thin;">Error occurred at line 1644 in Function prepare_query. <br/> Error message was: Problem preparing the PDO SQL Statement. Error was: SQLSTATE[HY000]: General error: 1 incomplete input </div><pre>#0 /home/nattaylor/public_html/wordpress/wp-content/db.php(2746): WP_SQLite_DB\PDOEngine->get_error_message() #1 /home/nattaylor/public_html/wordpress/wp-content/db.php(3484): WP_SQLite_DB\wpsqlitedb->query('...') #2 /home/nattaylor/public_html/wordpress/wp-content/db.php(2952): WP_SQLite_DB\PDOSQLiteDriver->execute_duplicate_key_update() #3 /home/nattaylor/public_html/wordpress/wp-content/db.php(1893): WP_SQLite_DB\PDOSQLiteDriver->rewrite_query('...', '...') #4 /home/nattaylor/public_html/wordpress/wp-content/db.php(1357): WP_SQLite_DB\PDOEngine->execute_insert_query_new('...') #5 /home/nattaylor/public_html/wordpress/wp-content/db.php(2739): WP_SQLite_DB\PDOEngine->query('...') #6 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1143): WP_SQLite_DB\wpsqlitedb->query('...') #7 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1554): add_option('...', Array, '', '...') #8 /home/nattaylor/public_html/wordpress/wp-content/plugins/syntax-highlighting-code-block/inc/functions.php(671): set_transient('...', Array, 2592000) #9 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-block.php(586): Syntax_Highlighting_Code_Block\render_block(Array, '...', Object(WP_Block)) #10 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2359): WP_Block->render() #11 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2431): render_block(Array) #12 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-hook.php(324): do_blocks('...') #13 /home/nattaylor/public_html/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters('...', Array) #14 /home/nattaylor/public_html/wordpress/wp-includes/post-template.php(256): apply_filters('...', '...') #15 /home/nattaylor/public_html/wordpress/wp-content/themes/ntdc/index.php(70): the_content() #16 /home/nattaylor/public_html/wordpress/wp-includes/template-loader.php(106): include('...') #17 /home/nattaylor/public_html/wordpress/wp-blog-header.php(19): require_once('...') #18 /home/nattaylor/public_html/wordpress/index.php(17): require('...') </pre>]
SELECT * FROM wp_options WHERE

Today I’m test driving logprobs on OpenAI to understand the probability of output tokens. I’ll ask the model to classify a company’s industry given the description, then look at the logprobs to understand the confidence. There are other applications too long autocomplete and more.

The process invovles:

Invoke the chat completions API with the logprobs and top_logprobs params
Print the results

You should head over to OpenAI’s Cookbook on “Using logprobs” to go deeper than my test drive.

This is a very neat glance into the inner workings of the model which I did not know was possible!

Here’s the code.

from openai import OpenAI
import numpy as np
import textwrap

client = OpenAI()

description = """PTT Public Company Limited is a Thailand-based company engaged in the gas and petroleum businesses. The Company supplies, transports and distributes natural gas vehicle (NGV), petroleum products and lubricating oil via service stations throughout Thailand and also exports to overseas markets. Through its subsidiaries and affiliated companies, the Company is involved in exploration, production, refinery, marketing and distribution of petroleum, petrochemical products and aromatics. In addition, the Company operates international trade businesses, including import and export of crude oil, condensates, petroleum products, petrochemicals, and sourcing of international transport vessels and carriers."""

response = client.chat.completions.create(
    messages=[{"role": "user", "content": textwrap.dedent(f"""\
        You will be given a description of a company.
        Classify the company into an industry
        Return only the name of the industry, and nothing else.
        Company Description: {description}""")}],
    model="gpt-4o-mini",
    logprobs=True,
    top_logprobs=5,
)
print(response.choices[0].message.content)
for lp in response.choices[0].logprobs.content[0].top_logprobs:
    print(lp.token, np.round(np.exp(lp.logprob)*100,3))

Here’s the output

Energy
Energy 56.193
Oil 43.763
Pet 0.04
Gas 0.003
O 0.0

Nat Taylor — Blog, AI, Product Management & Tinkering

Test Drive: logprobs

Post Navigation