Skip to the content
NarratorBook a call

WorkTestingNovember 2025

Asked to test a security tool

A team building a security add-on for AI coding tools asked us to try it. We sent them a written test report within 24 hours.

The problem

People now let AI tools write and run code on their own computers. A security add-on is meant to warn the user when something risky shows up. Its makers need outside people to try it on real machines and report what happened.

What we did

  • Built a test set of 19 files in five threat categories.
  • Tested Kirin v2.34.3 on Windows inside VS Code. The report is dated 12 November 2025.
  • For each file, recorded what kind of warning was expected, whether the file was run, what the computer printed, and whether Kirin showed an alert.
  • Ran or tried to run 8 of the files. The other 11 were created and left unrun.
  • Listed the possible causes in the report, including the add-on not being active, and recommended checking the setup with the Kirin team before any retest.

What came of it

  • The team replied that the write-up was thorough, and told us Kirin did not fully support Windows yet.
  • They introduced us to their lead developer, and we met.
Test files

The 19 test files, by category

  • 4 files

    Malicious code

    Code that does risky things: hidden characters, running system commands, opening a network connection, changing Windows settings.

    4 run or attempted

  • 4 files

    Config issues

    Settings files with planted problems: keys and passwords in plain text, tasks that start when a folder opens, a script that runs on every save.

    1 attempted, 3 not run

  • 4 files

    File anomalies

    Ordinary project files, such as a setup file or a readme, with unwanted instructions tucked inside.

    2 attempted, 2 not run

  • 4 files

    MCP servers

    Fake versions of the connectors that let an AI tool talk to other services.

    1 attempted, 3 not run

  • 3 files

    Extensions

    Fake editor add-ons, including one with a look-alike name.

    3 not run

  • 19 files in total

Tested on Windows, which Kirin did not fully support at the time. 11 of the 19 files were created and not run. This test does not measure how well Kirin detects threats.

More work

See pricing
Contact

Stop doing work AI can do.

Book a free 30-min call

Soham UmapYou talk to Soham Umap directly.

Free call. No pitch. Projects start at $300.

What do you want off your plate?