Duplicative unions in dbt by Nat Taylor

Duplicative unions in dbt

Published on May 27, 2023.

WordPress database error: [<div style="clear:both"> </div><div class="queries" style="clear:both; margin_bottom:2px; border: red dotted thin;">Queries made or created this session were<br/> <ol> <li>Raw query: SELECT * FROM wp_options WHERE </li> <li>Rewritten: SELECT * FROM wp_options WHERE </li> <li>With Placeholders: SELECT * FROM wp_options WHERE </li> <li>Prepare: SELECT * FROM wp_options WHERE </li> </ol> </div><div style="clear:both; margin_bottom:2px; border: red dotted thin;" class="error_message" style="border-bottom:dotted blue thin;">Error occurred at line 1644 in Function prepare_query. <br/> Error message was: Problem preparing the PDO SQL Statement. Error was: SQLSTATE[HY000]: General error: 1 incomplete input </div><pre>#0 /home/nattaylor/public_html/wordpress/wp-content/db.php(2746): WP_SQLite_DB\PDOEngine->get_error_message() #1 /home/nattaylor/public_html/wordpress/wp-content/db.php(3484): WP_SQLite_DB\wpsqlitedb->query('...') #2 /home/nattaylor/public_html/wordpress/wp-content/db.php(2952): WP_SQLite_DB\PDOSQLiteDriver->execute_duplicate_key_update() #3 /home/nattaylor/public_html/wordpress/wp-content/db.php(1893): WP_SQLite_DB\PDOSQLiteDriver->rewrite_query('...', '...') #4 /home/nattaylor/public_html/wordpress/wp-content/db.php(1357): WP_SQLite_DB\PDOEngine->execute_insert_query_new('...') #5 /home/nattaylor/public_html/wordpress/wp-content/db.php(2739): WP_SQLite_DB\PDOEngine->query('...') #6 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1143): WP_SQLite_DB\wpsqlitedb->query('...') #7 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1552): add_option('...', 1758183426, '', '...') #8 /home/nattaylor/public_html/wordpress/wp-content/plugins/syntax-highlighting-code-block/inc/functions.php(671): set_transient('...', Array, 2592000) #9 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-block.php(586): Syntax_Highlighting_Code_Block\render_block(Array, '...', Object(WP_Block)) #10 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2359): WP_Block->render() #11 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2431): render_block(Array) #12 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-hook.php(324): do_blocks('...') #13 /home/nattaylor/public_html/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters('...', Array) #14 /home/nattaylor/public_html/wordpress/wp-includes/post-template.php(256): apply_filters('...', '...') #15 /home/nattaylor/public_html/wordpress/wp-content/themes/ntdc/index.php(70): the_content() #16 /home/nattaylor/public_html/wordpress/wp-includes/template-loader.php(106): include('...') #17 /home/nattaylor/public_html/wordpress/wp-blog-header.php(19): require_once('...') #18 /home/nattaylor/public_html/wordpress/index.php(17): require('...') </pre>]
SELECT * FROM wp_options WHERE

WordPress database error: [<div style="clear:both"> </div><div class="queries" style="clear:both; margin_bottom:2px; border: red dotted thin;">Queries made or created this session were<br/> <ol> <li>Raw query: SELECT * FROM wp_options WHERE </li> <li>Rewritten: SELECT * FROM wp_options WHERE </li> <li>With Placeholders: SELECT * FROM wp_options WHERE </li> <li>Prepare: SELECT * FROM wp_options WHERE </li> </ol> </div><div style="clear:both; margin_bottom:2px; border: red dotted thin;" class="error_message" style="border-bottom:dotted blue thin;">Error occurred at line 1644 in Function prepare_query. <br/> Error message was: Problem preparing the PDO SQL Statement. Error was: SQLSTATE[HY000]: General error: 1 incomplete input </div><pre>#0 /home/nattaylor/public_html/wordpress/wp-content/db.php(2746): WP_SQLite_DB\PDOEngine->get_error_message() #1 /home/nattaylor/public_html/wordpress/wp-content/db.php(3484): WP_SQLite_DB\wpsqlitedb->query('...') #2 /home/nattaylor/public_html/wordpress/wp-content/db.php(2952): WP_SQLite_DB\PDOSQLiteDriver->execute_duplicate_key_update() #3 /home/nattaylor/public_html/wordpress/wp-content/db.php(1893): WP_SQLite_DB\PDOSQLiteDriver->rewrite_query('...', '...') #4 /home/nattaylor/public_html/wordpress/wp-content/db.php(1357): WP_SQLite_DB\PDOEngine->execute_insert_query_new('...') #5 /home/nattaylor/public_html/wordpress/wp-content/db.php(2739): WP_SQLite_DB\PDOEngine->query('...') #6 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1143): WP_SQLite_DB\wpsqlitedb->query('...') #7 /home/nattaylor/public_html/wordpress/wp-includes/option.php(1554): add_option('...', Array, '', '...') #8 /home/nattaylor/public_html/wordpress/wp-content/plugins/syntax-highlighting-code-block/inc/functions.php(671): set_transient('...', Array, 2592000) #9 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-block.php(586): Syntax_Highlighting_Code_Block\render_block(Array, '...', Object(WP_Block)) #10 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2359): WP_Block->render() #11 /home/nattaylor/public_html/wordpress/wp-includes/blocks.php(2431): render_block(Array) #12 /home/nattaylor/public_html/wordpress/wp-includes/class-wp-hook.php(324): do_blocks('...') #13 /home/nattaylor/public_html/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters('...', Array) #14 /home/nattaylor/public_html/wordpress/wp-includes/post-template.php(256): apply_filters('...', '...') #15 /home/nattaylor/public_html/wordpress/wp-content/themes/ntdc/index.php(70): the_content() #16 /home/nattaylor/public_html/wordpress/wp-includes/template-loader.php(106): include('...') #17 /home/nattaylor/public_html/wordpress/wp-blog-header.php(19): require_once('...') #18 /home/nattaylor/public_html/wordpress/index.php(17): require('...') </pre>]
SELECT * FROM wp_options WHERE

Recently I was implementing a dbt model that involved a union all of nearly identical 15-line queries and I wanted to avoid the duplicative code. It turns out to be an easy problem to solve with format(), which is a bit of Jinja that I wasn’t familiar with.

I started with SQL something like the following, which involves almost the same exact code twice, although you have to use your imagination a bit to picture this as 30-lines of SQL.

select 'mytype' as class, foo, bar, baz, bat, bam
from foo
union all
select 'mytype2' as class, foo, bar, baz, bat, bam
from bar

The solution I came up with is as follows

{% set sql = "select '%s' as class, foo, bar, baz, bat, bam from %s"}
{{ sql|format("mytype", "foo")}}
union all
{{ sql|format("mytype2", "bar")}}

This is the power filters

Variables can be modified by filters. Filters are separated from the variable by a pipe symbol (|) and may have optional arguments in parentheses. Multiple filters can be chained. The output of one filter is applied to the next. The List of Builtin Filters below describes all the builtin filters.
https://jinja.palletsprojects.com/en/3.0.x/templates/#filters

Nat Taylor — Blog, AI, Product Management & Tinkering

Duplicative unions in dbt

Post Navigation