AI and bug bounties: notes from 68 minutes on Across the Pondcast

Tib3rius and Andy Swift put episode 45 of Across the Pondcast up on Friday, and I'm the guest: AI & Bug Bounties.

We recorded it at the end of July, before Hacker Summer Camp, and early on I said the answer to most of what they were asking would be completely different again in three months. That clock is still running, so read the below as a late-July snapshot rather than a current position.

It runs 68 minutes and wanders from how Bugcrowd started, through what AI has actually done to bug bounty, to education and lobster rolls (the context for the lobster rolls didn't make the recording, sorry). Salient points are below with timestamps against them, so you can jump to whichever bit you came for. The times are matched against the published video's own caption track and start a few seconds early on purpose, so you land before the sentence rather than in the middle of it.

3:28 The math was wrong

I'd ended up running a pentest company, and the thing that bugged me was one person, paid by the hour, being asked to outsmart a crowd of adversaries with every kind of skill set and a lot more incentive to succeed. It doesn't really matter how smart or dumb that person is. Meanwhile the hacker community was sitting there waiting for an invite to the table, unplugged from the problem because everyone was scared of them, which seemed like a stupid problem for the internet to have.

5:51 It's a minor miracle the internet still works

The internet is way more vulnerable than most people realize or care to admit. What I feel AI has done is take the idea that the vulnpocalypse was already here, just not evenly distributed, and get everyone thinking about it all at once.

7:12 Your trusted pentester is also a hacker

The early objection was some version of "we're trying to keep those people out, which is why we only work with our trusted pentester"... who also, by the way, happens to be a hacker. None of it was rational when you thought it through, so a lot of early Bugcrowd was just education. The bad guys aren't waiting for an invite.

8:30 A swag company that dabbles in cybersecurity

That was the running joke for a while, and it was pretty deliberate. T-shirts, stickers and memes are fun for the lols on their own, and they also gave hackers who felt like outcasts a rally point to form a tribe around. I don't think we take credit for that community, it was already on its way. We saw our role as making it as easy and as fast as possible.

13:58 You must be this tall to ride

The thing that's a bit different about AI is that the height bar has dropped at the same time as the ability to discover has gone up. The cost of finding a legitimate issue is lower, the barrier to entry is lower, and that nets out to a whole lot more being discovered all at once while everyone simultaneously freaks out about it.

14:39 People were photoshopping XSS in 2012

Fake screenshots to game a payout are not new. We create games for people to break things, so one of the things they're going to try to break is the game itself. The spam and the legitimate findings have both been multiplied, and you end up with an enormous gap between the slop and the question of what to do with the increased rate of valid stuff.

16:00 Downgrading to a VDP is probably the most sane reaction

Some companies have shut the program down altogether, others have gone from an incentivized bounty to a plain vulnerability disclosure program. If you feel like you need to do something, I think that second one is the sane version. I'm all for bounty hunters getting paid, but it has to work as well. The really smart companies are thinking through what risk actually means to them and putting a greater incentive on the things they care about.

17:18 The dirty little secret

We were never going to get around to fixing all of it. That's been true this entire time, people just haven't been saying it out loud. Software is hard and we've gone off and created a lot of it. Even if there's some utopian future where code comes out perfect, there's a hell of a hump in the road between here and there, so prioritization it is.

20:45 You getting paid is not the point of the exercise

This can be a really difficult thing to explain to bug hunters. The point is making the software and the system more resilient, and you're a player in that story. A company using AI on its own code before it hits production is kind of the goal. Life is meant to get harder for us if we're doing this right.

23:41 Undocumented access feature enthusiasts

We kind of like being special, the ones who know the dark arts and all that. I've definitely seen more hunters burning out: reports not taken seriously, imposter syndrome kicking in, and it gets discouraging when standing out gets harder. It's not a new thing, but I do think it's gone up a lot with AI dropping itself into knowledge work. I think a lot about how you keep that pipeline going and keep the dopamine hit available to people coming in.

25:45 So did the steam engine

One of the hosts made the point that AI is doing as much damage as good at the moment. I think every transformational advance in technology has that quality. When I first saw generative AI it looked to me like the internet in about 1997: an enormous hype cycle, a whole bunch of BS, sorting the wheat from the chaff, and at the end of it something that touches pretty much everything. Probably the worst thing a security practitioner can do right now is be a bit of a Luddite about it.

27:21 The experts get the most out of it

In my experience the folks who are really strong in their niche end up the most proficient with AI, because they've got the subject matter expertise to steer it. They aren't delegating their brain out to it, they're amplifying what they already know. Same goes for people at the coalface of a problem who never saw themselves as builders and can now whip together a tool that at the very least communicates the concept. I want to be careful not to oversell that, it's not a silver bullet.

31:38 I ideate, Damien makes it real

The disclose.io lookup tool sat in the back of my brain forever: you're on internet safari, you find a broken thing, you work out who owns it and who to tell, and you always get at least a backstop answer. I know how to build things to break other things, but I'm not a platform engineer by any stretch of the imagination. Damien Radford, another former Bugcrowder, does the sanity, engineering and architecture passes. People still matter, I guess is the point.

33:10 No robots in the gym

That one is Daniel Miessler's, and I love it. You don't go to the gym to avoid resistance, you go for the resistance. If your job is moving things from A to B on a factory floor, that's the perfect place for the robot. A lot of it comes back to what you're uniquely good at, what you want to get better at, and whether this stuff becomes supporting infrastructure for that rather than a replacement.

35:35 Probabilistic by design

On whether prompt injection ever really gets solved: an LLM is probabilistic by design, so building deterministic rules into it means fighting the fundamental architecture of the thing you're trying to secure. Anyone who's worked in security for a while knows that's a thing that works until it doesn't. There's an entire class of attacks on AI systems we haven't really scratched the paint off yet, which to me is a lot of runway for research.

36:27 Not everything gets solved with a pull request

Auditing source and automatically patching libraries is great. Then you have to deploy and regression test that patch into thousands of products running in the wild, and get it actually implemented, and that's the top few turtles in a decades-old stack of turtles powering the internet. Even if we theoretically get all of it right, the bad guys aren't going to pack up and go home.

40:38 More of a graph than a list

A lot of what AI is being applied to on the fix side right now is very atomic: this vulnerability in this library. The really bad stuff is exploit chains, and chains are about the relationships between primitives at the app, network and architecture level. We haven't even started on AI that can assess that holistically, because we're still so focused on code, which is probably where we should be focused. There's a long tail to this.

42:09 Stupid decisions, faster, with more energy

In the meantime everyone is using this stuff to make stupid decisions faster with more energy, and you could argue pretty convincingly that we're adding to the problem in this particular season. We won't necessarily realize it until things start hitting the fan a bit later on. I'm clearly massively conflicted in saying this, and that's for a reason: attack and defense are both products of human incentive and human creativity, and that predates the internet as a concept.

47:17 Go where other people aren't looking

A lot of the advice I was giving researchers was to apply your focus to areas other people aren't necessarily thinking about. Everyone is looking at open source code, which is great. What about hardware? What about cryptosystems? The valuable question is what the frontier labs and the commercial tooling are going to take a while to get around to that is still relevant from a risk standpoint.

49:05 The baby and the bathwater

Organizations deciding to put their head in the sand because the outside world is too annoying was a very real thing when we recorded. The risk is that all the work done to establish the hacker as a good faith component of the security ecosystem goes out with the bathwater. If you build stuff you will have missed something, someone outside your organization will eventually find it and try to tell you, and you don't get to opt out of that. It's a function of the physics of the internet, and it's a lot of what we work on at disclose.io.

51:57 Then or now? Now

Asked whether I'd rather be pentesting back then or today, I went with now, honestly. For a lot of my time in the game pentesting has been a thing you sell to people who don't care, which is how you end up with the same vulnerabilities in the same test years later. I think that's becoming less of an option. When I'm doing security testing I'd like to see the thing get fixed.

54:46 The dog that caught the car

I landed in the US from Australia right around when Snowden did his thing, which to me was one of the first times the entire internet thought about security as potentially relevant to them, all at the same time. The upside is the ball got rolling. The downside is that I kind of miss just doing this for fun. We always wanted the relevance, and it's easy to be the dog that caught the car.

63:24 You can't teach an LLM wisdom

Knowledge, sure. Reasoning, sure. Wisdom is the thing you learn by screwing up, or by getting taught from other people's mistakes. Having heart surgery in the middle of all the Bugcrowd stuff gave me a lot of time to think about legacy and the cross-generational handoff, and I did a BSides Las Vegas keynote on it. In my head it's working when the younger lot can take advantage of these tools without getting skinned in the same ways we did.

65:51 Where I landed

Our role is to take assumptions, tip them upside down, see what falls out, and try to leave things better as a result. Technology doesn't ever really replace that, it gives us different things to play with and changes the playing field we're doing it on. As long as we stick with the important things I think everyone's going to be fine.

That's the lot. It got a fair bit more philosophical than any of us planned, which I apologized for on the recording and will again here. Thanks to Tib3rius and Andy for having me on.