Test Drive: OpenAI Evaluations
Published on .
Today I test drove OpenAI Evals, a tool to “regularly run evaluations (often called evals) on your model’s outputs using test data helps you build and maintain high-quality and reliable AI applications.” There are lots of eval tools out there, but it’s great to see something native within OpenAI. They make it fairly easy to build the dataset with the new “Stored Completions” functionality. The process is pretty simple:
- Upgrade to the latest OpenAI SDK
- Generate some completions with the
storeflag set to true - Go to https://platform.openai.com/evaluations and run a completion.
Here’s the test data I used:
from openai import OpenAI
client = OpenAI()
descriptions = """Cummins Inc. designs, manufactures, distributes and services diesel and natural gas engines and engine-related component products. The Company's segments include Engine, Distribution, Components and Power Systems. The Engine segment manufactures and markets a range of diesel and natural gas powered engines under the Cummins brand name, as well as certain customer brand names, for the heavy and medium-duty truck, bus, recreational vehicle (RV), light-duty automotive and agricultural markets. The Distribution segment consists of the product lines, which service and/or distribute a range of products and services, including parts, engines, power generation and service. The Components segment supplies products, including aftertreatment systems, turbochargers, filtration products and fuel systems for commercial diesel applications. The Power Systems segment consists of businesses, including Power generation, Industrial and Generator technologies.
Rio Tinto plc is a mining and metals company. The Company's business is finding, mining and processing mineral resources. The Company's segments include Iron Ore, Aluminium, Copper & Diamonds, Energy & Minerals and Other Operations. The Company operates an iron ore business, supplying the global seaborne iron ore trade. Its Iron Ore product operations are located in the Pilbara region of Western Australia. The Aluminium business includes bauxite mines, alumina refineries and aluminum smelters. Its bauxite mines are located in Australia, Brazil and Guinea. The Copper & Diamonds segment has managed operations in Australia, Canada, Mongolia and the United States, and non-managed operations in Chile and Indonesia. The Energy & Minerals segment consists of mining, refining and marketing operations in over 10 countries, across six sectors: borates, coal, iron ore concentrate and pellets, salt, titanium dioxide and uranium.
Rio Tinto Limited (Rio Tinto) is a mining company. The Company is focused on finding, mining and processing of mineral resources. Its segments include Iron Ore, Aluminum, Copper & Diamonds, Energy & Minerals, and Other Operations. Its products include aluminum, copper, diamonds, gold, industrial minerals (borates, titanium dioxide and salt), iron ore, thermal and metallurgical coal, and uranium. The Iron Ore product group's operations are located in the Pilbara region of Western Australia. The Company's business includes bauxite mines, alumina refineries and a range of aluminum smelters. The Copper & Diamonds product group has managed operations in Australia, Canada, Mongolia and the United States, and non-managed operations in Chile and Indonesia. The Energy & Minerals has operations across six sectors: borates, coal, iron ore concentrate and pellets, salt, titanium dioxide and uranium.
The Royal Dutch Shell plc explores for crude oil and natural gas around the world, both in conventional fields and from sources, such as tight rock, shale and coal formations. The Company's segments include Integrated Gas, Upstream, Downstream and Corporate. The Integrated Gas segment is engaged in the liquefaction and transportation of gas and the conversion of natural gas to liquids to provide fuels and other products, as well as projects with an integrated activity, ranging from producing to commercializing gas. The Upstream segment includes the operations of Upstream, which is engaged in the exploration for and extraction of crude oil, natural gas and natural gas liquids, and the marketing and transportation of oil and gas, and Oil Sands, which is engaged in the extraction of bitumen from mined oil sands and conversion into synthetic crude oil. The Downstream segment is engaged in oil products and chemicals manufacturing, and marketing activities.
BHP Billiton Plc is a global resources company. The Company is a producer of various commodities, including iron ore, metallurgical coal, copper and uranium. Its segments include Petroleum, Copper, Iron Ore and Coal. The Petroleum segment is engaged in the exploration, development and production of oil and gas. The Copper segment is engaged in mining of copper, silver, lead, zinc, molybdenum, uranium and gold. The Iron Ore segment is engaged in mining of iron ore. The Coal segment is engaged in mining of metallurgical coal and thermal (energy) coal. Its businesses include Minerals Australia, Minerals Americas, Petroleum and Marketing. It extracts and processes minerals, oil and gas from its production operations located primarily in Australia and the Americas. It manages product distribution through its global logistics chain, including freight and pipeline transportation. It sells its products through direct supply agreements with its customers and on global commodity exchanges.
BHP Billiton Limited is a global resources company. The Company is a producer of various commodities, including iron ore, metallurgical coal, copper and uranium. Its segments include Petroleum, Copper, Iron Ore and Coal. The Petroleum segment is engaged in the exploration, development and production of oil and gas. The Copper segment is engaged in mining of copper, silver, lead, zinc, molybdenum, uranium and gold. The Iron Ore segment is engaged in mining of iron ore. The Coal segment is engaged in mining of metallurgical coal and thermal (energy) coal. Its businesses include Minerals Australia, Minerals Americas, Petroleum and Marketing. The Company extracts and processes minerals, oil and gas from its production operations located primarily in Australia and the Americas. The Company manages product distribution through its global logistics chain, including freight and pipeline transportation. Its businesses include Minerals Australia, Minerals Americas, Petroleum and Marketing.
Petroleo Brasileiro S.A.-Petrobras specializes in the oil, natural gas and energy industry. The Company is engaged in prospecting, drilling, refining, processing, trading and transporting crude oil from producing onshore and offshore oil fields and from shale or other rocks. Its segments include Exploration and Production, which covers the activities of exploration, development and production of crude oil, natural gas liquid and natural gas; Refining, Transportation and Marketing, which covers the refining, logistics, transport and trading of crude oil and oil products activities, exporting of ethanol, and extraction and processing of shale; Gas and Power, which is engaged in transportation and trading of natural gas produced in Brazil and imported natural gas; Biofuels, which covers the activities of production of biodiesel and its co-products, and ethanol-related activities; Distribution, which includes the activities of its subsidiary Petrobras Distribuidora S.A., and Corporate.
Total S.A. (Total) is an oil and gas company. The Company has three segments: an Upstream segment, including the activities of the exploration and production of hydrocarbons, and the activities of gas and power; a Refining & Chemicals segment constituting an industrial hub consisting of the activities of refining, petrochemicals and specialty chemicals, and also includes the activities of oil trading and shipping, and a Marketing & Services segment, including the activities of supply and marketing in the field of petroleum products, as well as the activity of New Energies. Its Corporate segment includes holdings operating and financial activities. The Company operates in the renewable energies and power generation sectors. It is engaged in various sectors of oil and gas industry, including upstream (hydrocarbon exploration, development and production) and downstream (refining, petrochemicals, specialty chemicals, trading and shipping of crude oil and petroleum products and marketing).
Toyota Motor Corporation (Toyota) conducts business in the automotive industry. The Company also conducts business in finance and other industries. The Company's segments include Automotive, Financial Services and All Other. Toyota sells its vehicles in approximately 190 countries and regions. Toyota's markets for its automobiles are Japan, North America, Europe and Asia. The Company's Automotive segment includes the design, manufacture, assembly and sale of passenger vehicles, minivans and commercial vehicles, such as trucks and related parts and accessories. The Company's Financial Services segment consists of providing financing to dealers and their customers for the purchase or lease of Toyota vehicles. The All Other segment includes the design, manufacturing and sale of housing, telecommunications and other businesses. Its information technology related businesses include a Web portal for automobile information called GAZOO.com.
BP p.l.c. is an integrated oil and gas company. The Company owns an interest in OJSC Oil Company Rosneft (Rosneft), an oil and gas company. The Company's segments include Upstream, Downstream, Rosneft, and Other businesses and corporate. The Upstream segment is engaged in oil and natural gas exploration, field development and production, as well as midstream transportation, storage and processing. The Downstream segment has global manufacturing and marketing operations. The Rosneft segment has a resource base of hydrocarbons onshore and offshore. The Other businesses and corporate segment comprises the biofuels and wind businesses, shipping and treasury functions, and corporate activities around the world. The Company provides its customers with fuel for transportation, energy for heat and light, lubricants to keep engines moving and the petrochemicals products used to make everyday items as diverse as paints, clothes and packaging.
Volkswagen AG is engaged in developing vehicles and components for its brands. It also produces and sells vehicles, in particular passenger cars and light commercial vehicles for the Volkswagen Passenger Cars and Volkswagen Commercial Vehicles brands. The Passenger Cars segment cover the development of vehicles and engines, the production and sale of passenger cars, and the corresponding genuine parts business. The Commercial Vehicles segment comprises the development, production and sale of light commercial vehicles, trucks and buses, the genuine parts business and related services. The Power Engineering segment consist of the development and production of large-bore diesel engines, turbo compressors, industrial turbines and chemical reactor systems, the production of gear units, propulsion components and testing systems. The Financial Services segment comprises dealer and customer financing, leasing, banking and insurance activities, fleet management and mobility services.
Glencore plc is an integrated producer and marketer of commodities, such as metals and minerals, energy products, agricultural products and Corporate and other. The Metals and minerals segment is engaged in copper, zinc/lead, nickel, ferroalloys, alumina/aluminum and iron ore production and marketing. It also has interests in industrial assets that include mining, smelting, refining and warehousing operations. Its Energy products segment includes coal mining and oil production operations and investments in strategic handling, storage and freight equipment and facilities. Its Agricultural products segment is supported by controlled and non-controlled storage, handling and processing facilities in various locations, and is focused on grains, oils/oilseeds, cotton and sugar. Its diversified operations consist of over 150 mining and metallurgical, oil production and agricultural assets.
General Motors Company designs, builds and sells cars, trucks, crossovers and automobile parts. The Company's segments include GM North America (GMNA), GM Europe (GME), GM International Operations (GMIO), GM South America (GMSA) and General Motors Financial Company, Inc. (GM Financial). The Company provides automotive financing services through General Motors Financial Company, Inc. The Company develops, manufactures and/or markets vehicles in North America under the brands, including Buick, Cadillac, Chevrolet and GMC. The Company also develops, manufactures and/or markets vehicles outside North America under the brands, including Buick, Cadillac, Chevrolet, GMC, Holden, Opel and Vauxhall. The Company offers a range of after-sale vehicle services and products through the dealer network, such as maintenance, light repairs, collision repairs, vehicle accessories and extended service warranties. GM Financial is an automotive finance company, which provides automobile finance solutions.
Vale S.A. is a global producer of iron ore and iron ore pellets, key raw materials for steelmaking, and producer of nickel. The Company also produces copper, metallurgical and thermal coal, potash, phosphates and other fertilizer nutrients, manganese ore, ferroalloys, platinum group metals, gold, silver and cobalt. The Company's segments include Ferrous minerals, which comprises the production and extraction of ferrous minerals, as iron ore fines, iron ore pellets and its logistic services, manganese and ferroalloys and others ferrous products and services; Coal, which comprises the extraction of metallurgical and thermal coal and its logistic services; Base metals, which includes the production and extraction of non-ferrous minerals, and are presented as nickel and its byproducts, and copper (copper concentrated), and Others, which comprises sales and expenses of other products, services and investments in joint ventures and associate in other business.
PT Vale Indonesia Tbk is an Indonesia-based company primarily engaged in nickel mining and producing. It has nickel mining concessions in several areas in Sulawesi, Indonesia, including Kolonodale, Bahodopi, Sorowako-Towuti, Matano, Pomalaa and Suasua. The Company produces nickel in matte from lateritic ores at its integrated mining and processing facilities near Sorowako, Indonesia.
Honda Motor Co., Ltd. (Honda) develops, manufactures and markets motorcycles, automobiles and power products across the world. The Company's segments include Motorcycle Business, Automobile business, Financial services business, and Power product and other businesses. The Company produces a range of motorcycles, with engine displacement ranging from the 50 cubic centimeters class to the 1,800 cubic centimeters class. Its automobiles use gasoline engines of three, four or six cylinder, diesel engines, gasoline-electric hybrid systems and gasoline-electric plug-in hybrid systems. Honda offers a range of financial services to its customers and dealers through finance subsidiaries in countries, including Japan, the United States, Canada, the United Kingdom, Germany, Brazil and Thailand. Honda manufactures a range of power products, including general-purpose engines, generators, water pumps, lawn mowers, riding mowers, grass cutters, brush cutters, tillers and snow blowers.
Statoil ASA (Statoil) is an energy company. The Company is engaged in oil and gas exploration and production activities. The Company's segments include Development and Production Norway (DPN), Development and Production International (DPI), Marketing, Midstream and Processing (MMP) and Other. DPN segment manages the Company's upstream activities on the Norwegian continental shelf (NCS) and explores for and extracts crude oil, natural gas and natural gas liquids. DPI segment manages the Company's upstream activities that are not included in the DPN and Development and Production USA (DPUSA) business areas. MMP segment manages its marketing and trading activities related to oil products and natural gas, transportation, processing and manufacturing, and the development of oil and gas. Other segment includes activities in New Energy Solutions (NES), Technology, Projects and Drilling (TPD), Global Strategy and Business Development (GSB), and Corporate staffs and support functions.
Eni SpA (Eni) is an Italy-based company engaged in the exploration, development and production of hydrocarbons, in the supply and marketing of gas, liquefied natural gas (LNG) and power, in the refining and marketing of petroleum products, in the production and marketing of basic petrochemicals, plastics and elastomers and in commodity trading. The Company's segments include Exploration & Production, Gas & Power, and Refining & Marketing. Its Exploration & Production segment engages in oil and natural gas exploration and field development and production, as well as LNG operations in over 40 countries, including Italy, Libya, Egypt, Norway, the United Kingdom, Angola, Congo, Nigeria, the United States, Kazakhstan, Algeria, Australia, Venezuela, Iraq, Ghana and Mozambique. Its Gas & Power segment engages in supply, trading and marketing of gas, LNG and electricity, international gas transport activities and commodity trading and derivatives.
Engie SA, formerly GDF Suez SA, is a France-based natural gas and electricity supplier. Its operations are organized in five business lines: Energy Europe, engaged in the production of electricity and distribution and supplying of gas in continental Europe; Energy International which supplies power within North and Latin America, the United Kingdom, Turkey, Middle East, Asia and Africa; Global Gas & LNG, which includes exploration and production of gas and oil, procurement and routing of gas and Liquefied Natural Gas (LNG) and supplying accounts in Europe; Infrastructures, which operates the transport, supply and storage of natural gas; and Energy Services, providing multi-technical services in the areas of engineering, installation or energy services. The Company operates through La Compagnie du Vent, CNN MCO, which manages of all types of vessels, Siradel SAS and Green Charge Networks LLC, a Santa Clara-based manufacturer of energy storage systems and EV-Box BV, among others.""".split("\n")
def classify(desc):
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Determine the full GICS sub-industry code based on the description. Respond only with the code"},
{"role": "user", "content": desc}
],
store=True,
metadata={
"role": "classifier",
"department": "accounting",
"source": "testing"
}
)
return completion.choices[0].message
for desc in descriptions:
print(classify(desc))
