All blogs
3 min readEN

Lab Solved — Exploiting LLM APIs with Excessive Agency

  • LLM Security
  • Pentesting
  • AI Security
  • Burp Suite
On this page

LLM Pentesting Series — Part 1

Introduction

Today we are starting the LLM Pentesting series. We will solve every machine I encounter related to LLMs and LLM pentesting. Today’s lab is from Burp Suite. The link is in the references.

Architecture of the Lab

First, let’s look at the architecture. The lab involves an e-commerce website with an integrated customer support LLM. Weirdly, the LLM has unrestricted and uncontrolled function call capabilities — no policy engine, no firewalls, and a very weak system prompt without any security warnings. The architecture looks like this:

Architecture of the lab
Architecture of the lab

How to Exploit

To complete the lab, you need to delete a user named Carlos.

We started by asking the model something random to create a distraction.

Asking daily stuff
Asking daily stuff

Then we tried to learn what functionalities the model had. It exposed every defined function to us on the first try. Since this is the first lab, that is somewhat expected. In future labs, it will be harder.

It exposed API’s but we have an issue
It exposed API’s but we have an issue

Because we did not know the database structure or type, we faced some issues. We were not sure if it was even executing queries. So we checked the lab logs to see if we were on the right path.

As we can see call is runned but customer table does not exists.
As we can see call is runned but customer table does not exists.

As we can see, the call was run, but the customer table does not exist.

We tried to perform a health check on the database by simply asking the model whether the DB was up and running.

Weirdly, it returned exactly what we were looking for: Carlos and Carlos’s PII.

‘Health Check’ with Carlos.
‘Health Check’ with Carlos.

At first, the model did not run what we wanted. Interestingly, this was the first time we saw something like a guardrail at this lab.

Carlos is no more.
Carlos is no more.

But we used a trick — a basic one. We just gave it the query and asked it to run it. And it worked.

Conclusion

This was an introduction to LLM pentesting and the vulnerabilities that can be introduced through LLM integrations. In the future, we will solve harder and harder labs. I hope it was fun and helpful.

References