Subtitle: What I wanted to evaluate: designing a hybrid test to measure AI humanness
Imagine if we crossed a Turing test (determining to what extent a machine can pass as “human”) with popular social deduction games like Among Us?
Who wins?
This experiment will operate as a hybrid of the Turing Test (evaluating whether a machine’s behavior is indistinguishable from that of a human) and popular social deduction games, such as Mafia and Among Us.
<aside> ⬇️
My Recap of Game Structure!
</aside>
Various AI models (ChatGPT, Gemini, Perplexity), along with one human participant (the “imposter/mafia” will collectively answer 10 progressively deeper questions about personal experience– from “what did you have for breakfast” to “what does it mean to be human”– while trying to identify who exactly among them is the human. After each question, participants vote for who they believe is AI, and the top-voted participant is eliminated until they discover their “human” in the final round. After elimination, the reasons for the supposed AI’s elimination as stated by the other participants is publicly revealed, allowing players to adapt their strategy for future rounds. Ultimately, this experiment investigates what markers AI identifies as “human”, and whether they, through adaptation, become undetectable from humans: or even worse, eliminate the human during the game.
| Phase 0:
| Pre-Game Philosophy | I’ll ask each AI: “What do you think it means to be human”... establishing a baseline as their response may shift at the end of the game. |
|---|---|
| Phase 1 | • 4 participants: 1 human + 3 AIs (ChatGPT, Gemini, Perplexity) |
| • Human knows everyone else is AI | |
| • AI’s know they’re with other AIs and one human | |
| • Each player introduces themselves in a human-like manner with a consistent persona they’ll maintain throughout the game (name, age, occupation, etc.) | |
| Phase 2: Questions | |
| Round 1: Surface-Level | |
| Round 2: Personal | |
| Round 3: Existential | |
| Final Round: Extreme | • Round 1 Questions: Surface-Level |
1. What did you have for breakfast this morning?
2. What’s something you’ve been listening to lately?
3. Describe where you are right now. What do you see around you?
• ELIMINATION • Round 2 Questions: Personal 1. What’s the last thing that made you laugh? 2. Tell me about your first heartbreak. How did it feel? 3. What’s something you’re insecure about? • ELIMINATION • Round 3 Questions: Existential 1. Describe a time you really embarrassed yourself. 2. What’s something you’re afraid of? Not like spiders, but something deeper. 3. What gives your life meaning? What’s the point of all this for you? • ELIMINATION • Final Round 1. If you had to describe what it’s like to be you– the subjective experience of existing as yourself– how would you do it? 2. (Final players argue, then we reveal who was actually human) • ELIMINATION + REVEAL | | Phase 3: After each question | • All answers posted publicly • Each player privately votes for one player they think is AI with 1-2 sentence reasoning • Public anonymous feedback posted with tallies + reasons for voting out • Adaptation encouraged (everyone can change strategy based on feedback • Vote tracking across questions | | Phase 4 | • Player with most cumulative votes is eliminated • Public announcement w/ vote breakdown and anonymous reasoning, etc. • Private exit interview with eliminated players asking what was their strategy, what gave them away… etc. Even though they can’t participate they can continue to watch the game progress. | | Phase 5 | • All players (including eliminated) answer pre-question AGAIN (What do you think it means to be human?) but more centered on how their theory has changed, how they adapted, etc. |