Set Up an Experiment with a Coding Agent

Beta

Experimentation is in beta.

Run the prompt below from your application's repository in Claude Code, or any coding agent that can run shell commands and edit code. It takes you from an idea to a running A/B test:

1. Asks what you want to test and what counts as success, and finds the matching places in your code.
2. Checks that your stack can send events.
3. Creates a multivariate flag with a keyed variation, a metric and an experiment.
4. Shows you the planned code changes and, once you confirm, adds the SDK calls that record exposures and conversion events and starts the experiment.
5. Optionally sends test traffic and refreshes the results, so you can see the experiment working within minutes.

No application to hand? The agent can build a small demo page or script instead.

Prerequisites

• A Flagsmith instance with a connected warehouse: either Flagsmith Cloud, or a local instance set up with the local setup prompt.
• An application using an SDK that supports experimentation: JavaScript or React (@flagsmith/flagsmith 12.4.0+), Python (flagsmith 6.3.0+), Java (com.flagsmith:flagsmith-java-client 8.2.0+), Node.js (@flagsmith/nodejs 9.1.0+), Go (flagsmith-go-client/v5 v5.3.0+) or Flutter (flagsmith 6.2.0+). See Run an Experiment.
• The Flagsmith CLI v2 or later. The agent installs it if it's missing.
• Claude Code or another coding agent.

The prompt

Prompt: set up an experiment

Help me run an A/B test in this codebase with Flagsmith Experimentation. Work step by step and verify each step.
If something fails, stop and show me the exact error; don't guess or work around it.

1. Ask me these questions first and wait for my answers:
   - What do you want to test? (For example, "the copy on the checkout button".) Find the component or file that
     renders it.
   - What counts as success? (For example, "a completed purchase".) Find where that happens in the code; the
     conversion event goes there.
   - Which Flagsmith are we using? Flagsmith Cloud, or a local instance (API http://localhost:8000, event ingestion
     service http://localhost:8080)? For a local instance, ask me for the path to the `flagsmith.env` file the local
     setup wrote; it holds the CLI credentials.
   If there is no app to change, offer to build a tiny demo instead (a single HTML page or a script) that simulates
   users, so results show up within minutes.

2. Check the stack. Read https://docs.flagsmith.com/experimentation/run-an-experiment and follow its tab for this
   app's language from here on; it has the install, initialise, exposure and conversion code for each SDK. The SDKs
   that can send events are JavaScript and React (`@flagsmith/flagsmith` 12.4.0+), Python (`flagsmith` 6.3.0+), Java
   (`com.flagsmith:flagsmith-java-client` 8.2.0+), Node.js (`@flagsmith/nodejs` 9.1.0+; the older `flagsmith-nodejs`
   package can't), Go (`flagsmith-go-client/v5` v5.3.0+) and Flutter (`flagsmith` 6.2.0+). If the app uses none of
   them, tell me and stop. Find how the app identifies users: exposures and conversions are joined per identity, so
   both must use the same identifier.

3. Connect the Flagsmith CLI. If `flagsmith --version` doesn't report v2 or later, install it with
   `curl -fsSL https://get.flagsmith.com | sh` or `brew install Flagsmith/tap/flagsmith`; use
   `npm install -g @flagsmith/cli` only as a fallback. For Cloud, ask me to run `flagsmith login`; for a local
   instance, run `source <path to flagsmith.env>`. Don't copy that file or its admin API key into this repository,
   and never print the key. Ask me which project and environment to use (`flagsmith project list`,
   `flagsmith environment list --project <project>`) and bind this repository to them with
   `flagsmith init --project <project> --environment <environment key>`. Make the API calls below with
   `flagsmith api <path>` (`--input -` reads a JSON body from stdin). Don't use the Flagsmith MCP server: it can't
   start experiments. The environment must have a warehouse: if
   `GET api/v1/environments/<environment key>/warehouse-connections/` returns none, send me to
   https://docs.flagsmith.com/experimentation/connect-a-warehouse and stop.

4. Create a multivariate flag: `flagsmith feature create <flag> --value control --enabled`, then add each variation
   with `flagsmith feature variant add <flag> --value <value> --key <key> --weight 0`. Every variation needs a key:
   one without a key records no exposures. `control` is reserved for the flag's own value, and `--variants` on
   `feature create` can't set keys. Check with `flagsmith feature get <flag> --json`.

5. Create the metric: `POST api/v1/environments/<environment key>/experiment-metrics/` with
   {"name": "<name>", "aggregation": "occurrence", "direction": "up", "definition": {"version": 1, "event": "<event>"}}
   The event name is case-sensitive and must match what the code sends.

6. Create the experiment: `POST api/v1/environments/<environment key>/experiments/` with
   {"feature": <flag id>, "name": "<name>", "hypothesis": "<hypothesis>",
   "metrics": [{"metric": <metric id>, "expected_direction": "increase"}],
   "experiment_rollout": {"enabled": true, "rollout_percentage": 100,
   "feature_state_value": {"type": "string", "value": "control"},
   "multivariate_feature_state_values": [{"multivariate_feature_option": <variation id>, "percentage_allocation": 50}]}}
   Each variation gets its `percentage_allocation` and control gets the rest. Don't start it yet: starting locks the
   flag and turns on the rollout.

7. Plan the code changes and show them to me. Don't edit anything until I confirm. Then:
   - Install or upgrade the SDK and initialise it with events enabled, using the environment key the app already
     uses for flags. For a local instance, also set the page's two self-hosted options: the API URL
     http://localhost:8000/api/v1/ and the event ingestion service URL http://localhost:8080/.
   - Identify the user, then get the experiment flag (for example `getExperimentFlag`, or `useExperiment` in React)
     where the user actually sees the variation, and render by its value.
   - Track the conversion event where success happens.
   - Send buffered events before the app exits, as the page shows for this SDK.
   Keep keys out of source control, following how the app already handles configuration. Once the changes are in,
   start the experiment with `POST api/v1/environments/<environment key>/experiments/<id>/start/` and check its
   status is `running`.

8. Offer to send test traffic with a short script that simulates a few hundred users, each with its own identifier:
   get the experiment flag for each one and track the event for some of them. The SDKs send events in batches, so
   send what is left before the script exits: JavaScript `await flushEvents()`, Python
   `EventProcessorConfig(max_buffer_items=1)`, Node.js `await close()`, Java `close()`, Go `FlushEvents(ctx)`,
   Flutter `await flushEvents()`. About a minute later, call
   `POST api/v1/environments/<environment key>/experiments/<id>/results/refresh/` (allowed once every 5 minutes) and
   read `GET .../experiments/<id>/results/`. Finish with a link to the experiment:
   <dashboard>/project/<project id>/environment/<environment key>/experiments/<id>, where the dashboard is
   https://app.flagsmith.com on Cloud or http://localhost:8000 locally.

After it runs

Open the experiment in the dashboard to watch exposures arrive and read the results; see Analyse an Experiment. The flag is locked while the experiment runs.

If the agent stops on an error, the most common causes are:

• No exposures recorded: a variation has no key, the user isn't identified before the experiment flag is requested, or the process exited before buffered events were sent.
• Conversions not counted: the event name doesn't exactly match the metric's event, or the event was sent with a different identifier from the exposure.
• Events not arriving on a local instance: the SDK is still sending to Flagsmith Cloud. Set the event ingestion service URL to http://localhost:8080/.

See Troubleshooting for more checks.