concertcrawl.
CRAWLER INFO · /BOT

This is our crawler.

If you're here, ConcertCrawl/1.0 turned up in your server logs and you followed the URL it left. Good. That's what it's for.

Concert Crawl is a Denver concert listing site. We read public event pages so someone deciding what to see this week can find every show in town in one place, with what a ticket actually costs. When we list your show, the ticket link points at you. See what we're building

Everything our crawler does is on this page, including how to make it stop.

01

How to recognize us

Every request we make carries this string. Nothing we run identifies itself any other way. If you see a different agent claiming to be us, it isn't.

User-Agent
ConcertCrawl/1.0 (+https://concertcrawl.app/bot)
JavaScript
We render pages in a headless browser (Chromium) because many venue calendars only exist after JavaScript runs. We don't click, submit forms, or log in.
02

robots.txt

We read it before we read anything else.

Disallow
We request robots.txt before crawling and obey any Disallow that applies to us. If it's unreachable, we proceed at our normal rate.
Re-check
At the start of every pass, so at least once a day. An edit you make this afternoon is in force by tonight's run.
Crawl-delay
Honored. Where you state one, we take the longer of your value and our own 3-second default. We never speed up because a site permits it.
03

Rate and schedule

We're a small crawler on a slow clock. If we're still costing you more than you want to spend, section 5 is a two-line fix.

Request rate
We wait at least 3 seconds between requests, plus a random jitter so we never arrive on a fixed beat.
Per venue
That interval is configurable per site. If 3 seconds is still too fast for your box office, name a number and we'll set it for you.
Full pass
Once a day at 09:00 UTC.
Trigger checks
Between daily passes, the only thing that re-reads your site is one of us queueing your venue by hand. A job picks that queue up every 10 minutes. Nothing re-reads you automatically.
04

What we take, and what we don't

We read
Public event pages only: show name, performing artists, date and time, ticket price, the event description and image you publish, and the link back to your own ticketing page.
We never
Touch anything behind a login, submit a form, follow a checkout, or collect anything about a person. No customer data, no email addresses, no accounts. We don't have a login to your site and don't want one.
Retention
We keep a show after the date passes rather than deleting it, so people can look back at what they saw. There is no automatic purge, so assume we hold it until you ask us not to. Section 5 is how you ask.
Your page
Your venue's listings and price history are visible on one page:concertcrawl.app/venues/your-venueVenue pages are invite-only during our private beta and open to everyone at launch.
05

Making us stop

Two ways. The first needs nothing from us.

Add us to robots.txt

User-agent: ConcertCrawl Disallow: /

In force by the next pass, within 24 hours and usually sooner.

Or tell a person

venues@concertcrawl.app
A human reads this address, not a queue. Say the word and we'll stop crawling you. We answer within two business days.

Removing what we already have

Blocking the crawler stops new reads; it doesn't unpublish what we've already collected. Ask for removal in the same email and we'll take your venue's listings and price history down within 30 days, and write back when it's done. No account, no form, no conditions.

We'd rather be the link that sends someone to your box office than a line you have to block.