Making Lawns Everywhere Spookier with AI
Published on .
Wouldn’t it be fun to see your house with spookier Halloween decorations on the lawn? I thought so and I whipped something together to do just that. You can give it a try at https://apps.nattaylor.com/halloween/ (until my credits run out.) Here’s a demo of the Whitehouse lawn and an explanation of how I built it.
The main idea is to use an AI diffusion model inpaint a mask of the lawn. Here’s OpenAI on inpainting:
[Inpainting] allows you to edit or extend an image by uploading an image and mask indicating which areas should be replaced. The transparent areas of the mask indicate where the image should be edited, and the prompt should describe the full new image, not just the erased area.
https://platform.openai.com/docs/guides/images/edits-dall-e-2-only
I normally defer to OpenAI for quick prototypes, but for some reason I chose Stability AI, which also has an inpainting endpoint.
For masking, I had heard of Meta’s SegmentAnything (SAM) but I couldn’t find an easy hosted version. What I did find pretty quickly was Image Segmentation and I was literally amazed by how simple it was to implement nvidia/segformer-b1-finetuned-cityscapes-1024-1024 which has a class for “terrain” that worked great for lawns.
For getting the image, I chose to use the Google Street View API for its simplicity.
As always, there was a bunch of prompt engineering involved.
The program flow then turned out to be quite simple and I was able to whip something together in about hour.
+-----------+ +-----------+ +---------+
| | | | | |
| Get Image |---->| Mask Lawn |---->| Inpaint |
| | | | | |
+-----------+ +-----------+ +---------+As is often the case, deploying it turned out to be the long pole. I ran in to two main challenges: secrets and memory. Secrets are boring and I won’t go into the details, but I was just DoingItWrong™️ in Flask — first with not explicitly loading dotenv in wsgi.py and then loading .env from the wrong path. The memory issue was far more cryptic as the error was something about “truncated headers.” Anyway, the solution was actually to offload masking to StabilityAI by using their search and replace endpoint! The payload spec is delightfully clean and my implementation looked something like this:
requests.post(
f"https://api.stability.ai/v2beta/stable-image/edit/search-and-replace",
headers={
"authorization": f"Bearer {os.getenv('STABILITY_API_KEY')}",
"accept": "image/*"
},
files={
"image": image_content,
},
data={
"prompt": prompt,
"search_prompt": "lawn",
"output_format": "jpeg",
},
)I appreciated how simple and effective the search prompt is!
In the end, here’s what I came up with.
import requests
import logging
from PIL import Image
import requests
import io
from textwrap import dedent
import os
import tempfile
import logging
import sys
log = logging.getLogger('app.halloween')
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(logging.Formatter('%(name)s - %(levelname)s - %(message)s'))
log.addHandler(handler)
log.setLevel(logging.DEBUG)
def generate(form):
log.debug("Beginning processing")
tmp = tempfile.NamedTemporaryFile(delete=False)
f = open(tmp.name, 'wb')
lawn, ll = get_lawn(form['ll'])
mask = None
native = inpaint(lawn, mask, 'halloween3')
native.save(f, 'JPEG')
return tmp.name
def get_lawn(pos):
"""get a random lawn"""
image = requests.get(f"https://maps.googleapis.com/maps/api/streetview?size=512x512&location={pos}&radius=500&key={os.getenv('GMAPS_API_KEY')}&return_error_code=true")
geo = {'nearest': {'latt': pos.split(',')[0], 'longt':pos.split(',')[1]}}
im = Image.open(io.BytesIO(image.content))
logging.info(f"Location: {geo['nearest']['latt']},{geo['nearest']['longt']}")
return (im, (geo['nearest']['latt'],geo['nearest']['longt']))
def inpaint(image, mask, variant='english', prompt=None):
logging.info(f'Inpainting style is {variant}')
variants = {
'english_old': "tranquil English-style garden on a crisp spring morning, where the scent of roses mingles with the soft murmur of a nearby brook, and pathways lined with manicured hedges lead to a gazebo draped in climbing ivy under a canopy of ancient oak trees",
'english': "(English garden style landscaping)+++, featuring (well-trimmed shrubs)++, (colorful perennials)+++",
'native': "A naturalized landscape design is generally loose and flowing, with an emphasis on native plants, weathered stone and other natural elements.",
"modern": "A modern landscape design features straight, clean lines, geometric shapes, orderly plantings and clipped hedges. This style embraces the less is more",
"halloween": "A spooky halloween scene with lots of skeletons, pumpkins and gravestones",
"halloween2": dedent("""\
Think creatively lit pumpkins, friendly ghosts, and silly skeletons
alongside eerie spiderwebs, flickering lights, and maybe even a
tastefully placed tombstone or two. A well-decorated home will
create a welcoming yet spooky atmosphere that delights
trick-or-treaters and passersby alike, inviting them to
enjoy the spirit of the holiday."""),
"halloween3": dedent("""\
Halloween lawn decorations including creatively lit pumpkins,
skeletons emerging from the ground, spooky tombstones,
spooky giant spiders alongside eerie spiderwebs,
ghosts and a giant 12-foot skeleton."""),
}
if not prompt:
prompt = variants[variant]
with io.BytesIO() as output:
if not mask:
mask = Image.new('RGB', (512, 512), color='white')
mask.save(output, 'JPEG')
mask_content = output.getvalue()
with io.BytesIO() as output:
image.save(output, 'JPEG')
image_content = output.getvalue()
response = requests.post(
f"https://api.stability.ai/v2beta/stable-image/edit/search-and-replace",
headers={
"authorization": f"Bearer {os.getenv('STABILITY_API_KEY')}",
"accept": "image/*"
},
files={
"image": image_content,
},
data={
"prompt": prompt,
"search_prompt": "lawn",
"output_format": "jpeg",
},
)
if response.status_code == 200:
return Image.open(io.BytesIO(response.content))
else:
raise Exception(str(response.text))