[{"data":1,"prerenderedAt":456},["ShallowReactive",2],{"blog-guides\u002Fai-advisor-small-sample-limits":3},{"id":4,"title":5,"body":6,"category":426,"date":427,"dateModified":428,"description":429,"draft":430,"extension":431,"faq":432,"featured":430,"keywords":439,"meta":440,"navigation":441,"ogDescription":442,"ogTitle":443,"path":444,"readTime":445,"schemaType":446,"seo":447,"sitemap":448,"stem":449,"tags":450,"twitterCard":454,"__hash__":455},"blog\u002Fblog\u002Fguides\u002Fai-advisor-small-sample-limits.md","AI Advisor Limitations: What 200 Signups Can Prove",{"type":7,"value":8,"toc":416},"minimark",[9,13,19,24,27,30,41,44,50,54,57,63,82,87,108,111,115,118,124,127,130,140,143,360,366,376,382,388,391],[10,11,12],"p",{},"Two hundred signups is enough to run a launch and not enough to run a regression. The OperatorStack AI Advisor will still write you five bullet points about your referral program at that size, and some of those bullets are solid while others are one signup away from saying the opposite. The difference is not in how confident the writing sounds. It is in which number the bullet is built on.",[14,15,16],"tldr",{},[10,17,18],{},"At a few hundred signups, trust the AI Advisor on counts, queues, and presence or absence: how many signups, which messages are unanswered, whether errors are firing, whether the script is still reporting. Treat every percentage and every week-over-week comparison as provisional, because at that denominator a single new signup moves the number by a visible amount.",[20,21,23],"h2",{"id":22},"the-one-signup-test","The One-Signup Test",[10,25,26],{},"Before acting on any advisor bullet that contains a percentage, do this: add one to the numerator and recompute. If the number moves enough to change your decision, the bullet is describing your sample, not your product.",[10,28,29],{},"Referral share is the clearest case. The advisor computes it as:",[31,32,38],"pre",{"className":33,"code":35,"language":36,"meta":37},[34],"language-python","referral_pct = round(referral_signups \u002F total_signups * 100, 1)\n","python","",[39,40,35],"code",{"__ignoreMap":37},[10,42,43],{},"With 200 total contacts, every referral signup is worth 0.5 points of referral share. A card reading \"referrals are 4.5% of signups, down from 6%\" is describing a swing of three people. At 2,000 contacts the same three people move the number by 0.15 points and the trend means something. The formula did not change. The denominator did.",[45,46,47],"warning-box",{},[10,48,49],{},"The comparison the advisor draws against its own previous response makes this worse, not better. It is given your last analysis for the same section and asked to note whether things are improving or worsening. Two noisy readings produce a confident-sounding trend between them. \"Referral share is declining\" can be three people who happened to sign up through a direct link this week.",[20,51,53],{"id":52},"counts-survive-small-samples-rates-do-not","Counts Survive Small Samples, Rates Do Not",[10,55,56],{},"The advisor pulls nine sections, and they do not degrade equally as your data shrinks. Sort them by what they are actually measuring.",[10,58,59],{},[60,61,62],"strong",{},"Reliable at any size (counts and presence):",[64,65,66,70,73,76,79],"ul",{},[67,68,69],"li",{},"Total signups, 30-day signups, 7-day signups. These are facts.",[67,71,72],{},"Unanswered messages. A queue of 3 unanswered pricing questions is 3 real people waiting, whatever your scale.",[67,74,75],{},"Error signatures. One JavaScript error that breaks the signup form breaks it for everyone who hits that path. You do not need a second occurrence to justify fixing it.",[67,77,78],{},"Installation status. The script is either reporting or it is not.",[67,80,81],{},"Which pages exist and which ones got traffic at all.",[10,83,84],{},[60,85,86],{},"Provisional at small samples (rates and comparisons):",[64,88,89,92,95,98,105],{},[67,90,91],{},"Conversion rate. At 200 visitors, one signup is half a percentage point.",[67,93,94],{},"Referral share, as above.",[67,96,97],{},"Week-over-week change on any metric.",[67,99,100,101,104],{},"Message response rate, computed as ",[39,102,103],{},"responded \u002F len(messages)",". With 6 messages, each one is 16.7 points.",[67,106,107],{},"Best and worst day in the daily signup series.",[10,109,110],{},"That last one deserves its own section, because it is the bullet founders act on most and the one small samples distort hardest.",[20,112,114],{"id":113},"why-your-best-day-was-tuesday-is-usually-noise","Why \"Your Best Day Was Tuesday\" Is Usually Noise",[10,116,117],{},"The signups section hands the model a 30-day daily series, with missing days filled in as zeros:",[31,119,122],{"className":120,"code":121,"language":36,"meta":37},[34],"def _fill_daily(counts: dict[str, int], days: int = 30) -> dict[str, int]:\n    \"\"\"Ensure every day in the range has an entry (fill gaps with 0).\"\"\"\n    if not counts:\n        return counts\n    today = date.today()\n    filled: dict[str, int] = {}\n    for i in range(days - 1, -1, -1):\n        d = (today - timedelta(days=i)).isoformat()\n        filled[d] = counts.get(d, 0)\n    return filled\n",[39,123,121],{"__ignoreMap":37},[10,125,126],{},"Then it asks the model to identify the best and worst days and judge whether momentum is building or fading.",[10,128,129],{},"Two hundred signups spread across 30 days averages 6.7 a day. Real daily counts will land somewhere between 2 and 12 by ordinary variation alone, with no change in your marketing, your page, or anything else. A day with 11 and a day with 3 sit in the same series, and the model will faithfully report that your best day was nearly four times your worst.",[131,132,133],"info-box",{},[10,134,135,136,139],{},"The zero-fill is the right engineering choice and a trap for interpretation. It means a day you got no signups appears as a real ",[39,137,138],{},"0"," in the series rather than a gap, which keeps the trend math honest. It also means a quiet weekend produces two zeros that look dramatic next to a weekday 8, when the whole range is a handful of people either way.",[10,141,142],{},"What to take from the daily series at this size: the shape over weeks, not days. If weeks one through three run 60, 58, 62 and week four runs 12, that is worth a look. If Tuesday beat Thursday, that is a coin flip you are narrating.",[144,145,148,152,155,161,164,251,254,264,268,271,277,280,283,286,290,293,296,299,302,306,315,323,331,339,347,350],"stat-box",{"label":146,"number":147},"referral share moved by one signup at 200 contacts","0.5pt",[20,149,151],{"id":150},"the-performance-card-has-a-specific-small-sample-bug-you-should-know-about","The Performance Card Has a Specific Small-Sample Bug You Should Know About",[10,153,154],{},"This one is not a judgment call. The hourly performance breakdown computes a p75 like this:",[31,156,159],{"className":157,"code":158,"language":36,"meta":37},[34],"vals = sorted(hourly_lcp[hour_key])\nidx = int(len(vals) * 0.75)\np75 = vals[min(idx, len(vals) - 1)]\nhourly_lines.append(f\"  {hour_key}: LCP p75={p75:.0f}ms ({len(vals)} samples)\")\n",[39,160,158],{"__ignoreMap":37},[10,162,163],{},"Work the index for small hours:",[165,166,167,185],"table",{},[168,169,170],"thead",{},[171,172,173,177,182],"tr",{},[174,175,176],"th",{},"Samples in the hour",[174,178,179],{},[39,180,181],{},"int(len * 0.75)",[174,183,184],{},"What gets reported as p75",[186,187,188,200,210,220,230,240],"tbody",{},[171,189,190,194,197],{},[191,192,193],"td",{},"2",[191,195,196],{},"1",[191,198,199],{},"the slowest of the 2",[171,201,202,205,207],{},[191,203,204],{},"3",[191,206,193],{},[191,208,209],{},"the slowest of the 3",[171,211,212,215,217],{},[191,213,214],{},"4",[191,216,204],{},[191,218,219],{},"the slowest of the 4",[171,221,222,225,227],{},[191,223,224],{},"5",[191,226,204],{},[191,228,229],{},"second slowest",[171,231,232,235,238],{},[191,233,234],{},"8",[191,236,237],{},"6",[191,239,229],{},[171,241,242,245,248],{},[191,243,244],{},"20",[191,246,247],{},"15",[191,249,250],{},"a real p75",[10,252,253],{},"For any hour with 2, 3, or 4 pageviews, the \"p75\" is literally your single worst measurement that hour. One person on hotel wifi becomes an hourly LCP of 6,200ms, and the advisor reads that line as a performance problem.",[255,256,257],"tip-box",{},[10,258,259,260,263],{},"The sample count is printed right there in the line (",[39,261,262],{},"(4 samples)","), which is the only reason you can catch this. When the performance card flags an hourly LCP spike, check the sample count on that hour before you go optimize an image. Below roughly 10 samples an hour, use the daily performance score instead, which aggregates across the whole day.",[20,265,267],{"id":266},"a-missing-card-is-not-a-broken-card","A Missing Card Is Not a Broken Card",[10,269,270],{},"Every section gates itself on having any data at all and returns nothing otherwise:",[31,272,275],{"className":273,"code":274,"language":36,"meta":37},[34],"total_signups = total_result.scalar() or 0\n\nif total_signups == 0:\n    return None\n",[39,276,274],{"__ignoreMap":37},[10,278,279],{},"Forms do the same on two conditions, a zero form count and a zero submission count in the last 90 days. Chat returns nothing with no conversations. Analytics returns nothing when pageviews and unique visitors are both zero.",[10,281,282],{},"So a small project legitimately shows four cards instead of eight. That is the system declining to comment rather than failing, and it is the behavior you want: the alternative is a referral card generated from zero referrals, which is exactly the manufactured insight the system prompt forbids.",[10,284,285],{},"Note that the windows differ by section. Signups and referrals look back 30 days; forms look back 90. A form that collected submissions two months ago still produces a card after the signup trend has gone quiet.",[20,287,289],{"id":288},"what-the-advisor-already-does-to-protect-you","What the Advisor Already Does to Protect You",[10,291,292],{},"Two things in the prompt are doing real work at small scale, and knowing they exist tells you where the remaining gap is.",[10,294,295],{},"It is told to judge relative to your own scale, with an explicit example: a hobby site going from 5 visitors a day to 3 is not red, while 500 dropping to 50 is. So the color on a card is not an absolute threshold you can compare against another project.",[10,297,298],{},"It is also told that if nothing interesting stands out, it should say so briefly rather than invent something. A card that reads \"nothing much changed this week\" is the system working, not the system being lazy.",[10,300,301],{},"The gap is that nothing in the pipeline attaches a confidence interval to a rate. The ACTION field is capped at 180 characters and SUMMARY at 250, because both render into fixed-width dashboard components and truncate mid-sentence past those limits. There is no room in a 180-character action for \"though this is based on 11 signups and would reverse with two more.\" The hedge you want has nowhere to live, so you have to supply it.",[20,303,305],{"id":304},"what-to-actually-do-with-200-signups","What to Actually Do With 200 Signups",[307,308,309],"step",{"number":196},[10,310,311,314],{},[60,312,313],{},"Work the queues first."," Unanswered messages, unresolved chat conversations, open error signatures. These are correct at any sample size and they are the only cards where \"the advisor said so\" is sufficient reason to act.",[307,316,317],{"number":193},[10,318,319,322],{},[60,320,321],{},"Apply the one-signup test to every percentage."," Recompute with one more in the numerator. If your decision flips, note the number and move on without acting.",[307,324,325],{"number":204},[10,326,327,330],{},[60,328,329],{},"Read the daily signup series at week granularity."," Compare week totals, not days. Four weekly numbers from 200 signups are worth more than 30 daily ones.",[307,332,333],{"number":214},[10,334,335,338],{},[60,336,337],{},"Check sample counts before trusting any p75."," The performance card prints them. Below 10 per hour, drop to the daily score.",[307,340,341],{"number":224},[10,342,343,346],{},[60,344,345],{},"Write down the rate-based questions you are deferring."," Conversion rate by source, referral share, which headline won. These need a bigger denominator, and the list tells you what to re-examine at 1,000 signups instead of relitigating it weekly at 200.",[10,348,349],{},"The honest summary is that the advisor is a very good assistant for noticing things and a poor one for settling arguments, and 200 signups is squarely in the range where those two jobs look identical on screen. Counts, queues, and breakage are things to notice. Rates are arguments. Treat them differently.",[351,352,353],"faq-section",{},[354,355,357],"faq-item",{"question":356},"How many signups before the AI Advisor is actually useful?",[10,358,359],{},"It is useful from the first day for counts and presence checks: how many signups, which messages are unanswered, whether errors are firing, whether the script is still reporting. Rate-based reads (conversion percentage, referral share, week-over-week change) need a few hundred events before a one-unit change stops moving the number by a visible amount. At 200 contacts, one referral signup moves referral share by 0.5 points.",[354,361,363],{"question":362},"Why did the AI Advisor show fewer cards than last week?",[10,364,365],{},"Each of the nine sections has its own data gate and returns nothing when its underlying count is zero. No form submissions in the last 90 days means no forms card. No chat conversations means no chat card. A missing card is the advisor declining to comment, not an error, and it is the correct behavior at small scale.",[354,367,369],{"question":368},"Can I trust the performance card's LCP p75 with low traffic?",[10,370,371,372,375],{},"Not hourly. The p75 is computed as the value at index ",[39,373,374],{},"int(samples * 0.75)",", so an hour with 2, 3, or 4 pageviews reports your single slowest pageview as the p75, and 5 through 8 samples report the second slowest. Read the daily or weekly performance score instead, and ignore hour-to-hour swings until each hour carries double-digit samples.",[354,377,379],{"question":378},"Does the advisor know my sample is small?",[10,380,381],{},"Partly. The system prompt tells it to judge metrics relative to the project's own scale, so a drop from 5 visitors a day to 3 is not flagged red the way 500 to 50 would be. It is also told not to manufacture insights when nothing stands out. What it does not do is attach a confidence interval to a percentage, so you still have to apply the one-signup test yourself.",[354,383,385],{"question":384},"Should I wait to launch until I have enough data for the advisor?",[10,386,387],{},"No. The counts, queues, and error signatures that matter most before launch are reliable at any size. Waiting for statistical power on a conversion rate means waiting for traffic you only get by launching. Use the advisor for what is broken or unanswered now, and defer the rate-based decisions until the denominator is large enough to survive one more signup.",[10,389,390],{},"::",[392,393,394,395],"content-related-articles",{},"\n  ",[396,397,394,401],"contentrelatedcard",{"href":398,"title":399,"description":400},"\u002Fblog\u002Fguides\u002Fai-advisor-launch-readiness","AI Advisor Launch Readiness: What Red, Yellow, Green Mean","The mechanism behind the cards: the 24-hour wait, the nine sections, and what moves a task between colors.",[396,402,394,406],{"href":403,"title":404,"description":405},"\u002Fblog\u002Fhow-to\u002Fhow-long-to-run-ab-test","How Long to Run an A\u002FB Test Before You Call a Winner","The same denominator problem applied to testing: pick the sample size and the stopping rule before you start.",[396,407,411],{"href":408,"title":409,"description":410},"\u002Fblog\u002Fguides\u002Ffirst-week-analytics-data","First Week Analytics: How to Read a Nearly Empty Dashboard","What 28 visitors can tell you, and why your conversion rate moves in whole-digit jumps that early.",[412,413],"cta-box",{"href":414,"label":415},"\u002F","Get Started Free",{"title":37,"searchDepth":417,"depth":417,"links":418},2,[419,420,421,422,423,424,425],{"id":22,"depth":417,"text":23},{"id":52,"depth":417,"text":53},{"id":113,"depth":417,"text":114},{"id":150,"depth":417,"text":151},{"id":266,"depth":417,"text":267},{"id":288,"depth":417,"text":289},{"id":304,"depth":417,"text":305},"guides","2026-10-01",null,"At 200 signups the AI Advisor reads real counts but noisy rates. Here is which cards are load-bearing at that size and which ones are rounding error.",false,"md",[433,434,435,437,438],{"question":356,"answer":359},{"question":362,"answer":365},{"question":368,"answer":436},"Not hourly. The p75 is computed as the value at index int(samples * 0.75), so an hour with 2, 3, or 4 pageviews reports your single slowest pageview as the p75, and 5 through 8 samples report the second slowest. Read the daily or weekly performance score instead, and ignore hour-to-hour swings until each hour carries double-digit samples.",{"question":378,"answer":381},{"question":384,"answer":387},"ai advisor limitations,small sample size analytics,operatorstack ai advisor,waitlist sample size,early stage analytics limits,when to trust analytics data",{},true,"Which AI Advisor cards you can act on with a small list, and which percentages are one signup away from telling you something else.","AI Advisor Limitations at 200 Signups","\u002Fblog\u002Fguides\u002Fai-advisor-small-sample-limits","8 min","Article",{"title":5,"description":429},{"loc":444},"blog\u002Fguides\u002Fai-advisor-small-sample-limits",[451,452,453],"ai-advisor","analytics","sample-size","summary_large_image","j1zDuX3hJgAgF96qnDJfl8FTPmrYKomR4tuz4mUDtDI",1793197855444]