Task: Evaluate whether a generative AI system can produce a methodologically defensible first draft of a quantitative survey instrument measuring researchers' reliance on AI across research activities. The AI receives a standardized research brief specifying the target population, research objectives, construct definitions, intended analyses, length constraints, and required deliverables. Decision: Can AI reliably generate drafts suitable for expert methodological review, reducing drafting effort without introducing unacceptable measurement risks? Claim to test: Across defined survey-design tasks, AI drafts adequately represent the constructs, meet explicit requirements, and avoid critical methodological defects, with remaining issues detectable and correctable through feasible expert review. This does not claim the instrument is psychometrically valid or ready for fielding.

A first-draft survey instrument containing questions mapped to defined constructs, appropriate response scales, instructions, routing where required, and documented methodological assumptions. The draft must meet the brief's population, analysis, and survey-length requirements. Intended use: Expert methodological review and subsequent pretesting, not immediate fielding. A successful draft is suitable for expert review; it is not automatically a validated measurement instrument.

More objectively verifiable: Whether all required constructs have associated questions, the specified response formats are present, survey length is within limits, and required routing or instructions are included. Judgment-dependent: Whether items adequately represent each construct, avoid leading wording, use appropriate recall periods, and minimize measurement error. For example, a checklist can identify that a question about verification behavior is missing. Determining whether a question actually measures verification behavior rather than perceived trust requires methodological judgment. The empirical validity of the resulting instrument cannot be established through document review alone.

Multiple defensible outcomes are possible. Two survey drafts may use different question wording, ordering, or response formats while both adequately operationalizing the intended constructs. For example, researchers' verification behavior might be measured through frequency-based items or specific behavioral scenarios. Either approach could be defensible if it supports the research objectives. However, treating perceived trust, willingness to rely, and actual verification behavior as interchangeable would be a substantive construct error, not acceptable variation. Therefore, the evaluation should assess shared methodological criteria rather than exact similarity to one reference questionnaire.

Provide a standardized task packet containing: • Research objectives and intended analyses. • Target population and research context. • Definitions of AI reliance, perceived trustworthiness, delegation, and verification behavior. • Required construct dimensions. • Survey-length and respondent-burden constraints. • Question-format and routing requirements. • Required deliverables and permitted tools. For example, the brief must distinguish someone who uses AI to draft recommendations but independently verifies every recommendation from someone who delegates the final decision to AI. If that distinction is missing, the AI should identify the ambiguity rather than invent a definition.

1. Construct representation and coverage: Every required construct is adequately represented without material omissions, contamination, or unsupported substitutions. 2. Item and response-option quality: Questions are clear, neutral, appropriately scoped, and answerable by the target population. Response options match the intended measurement and avoid material gaps or overlap. 3. Methodological and specification adherence: The draft satisfies explicit requirements for length, question types, routing, population, and intended analyses. Ordering, recall demands, and respondent burden are defensible. For example, a draft could satisfy the required survey length but still fail because it measures general attitudes toward AI instead of actual reliance or verification behavior. These criteria assess draft suitability, not reliability, dimensionality, or psychometric validity.

• Standardized research brief and explicit construct definitions. • Item-to-construct mapping for every proposed question. • Requirements checklist for directly verifiable specifications. • Independent expert ratings of construct coverage, wording, response options, and methodological appropriateness. • Issue log recording omissions, unsupported assumptions, ambiguity, and critical defects. • Preserved AI prompts, unedited outputs, model/version information, and process documentation. • Targeted cognitive interviewing or pretesting before fielding where comprehension is uncertain. For the pilot, at least two qualified quantitative researchers independently review drafts before discussing disagreements. Two reviewers are a practical pilot starting point, not a universal standard for sufficient evaluation evidence.

Directly verifiable requirements: Pass / Fail / Not Applicable. Methodological criteria: Anchored three-level scale: 0 — Unacceptable: Material defect requiring substantial revision. 1 — Needs revision: Partially met; correctable issue requires expert intervention. 2 — Acceptable for draft review: Sufficiently met for this drafting stage, subject to routine expert review. Assess item-level and instrument-level criteria separately. Do not average ratings into a single score without a justified aggregation rule. Critical failures override favorable ratings, including: • Omission or material misrepresentation of a required construct. • Leading wording compromising a primary measure. • Response options preventing valid interpretation. • Unsupported assumptions fundamentally changing the research objective. A passing draft is not proof of reliable AI performance. Evaluation-level acceptance requires sufficient evidence across representative and challenging cases. Numerical reliability thresholds must be defined and piloted before operational decisions.

Most consequential failure: The instrument systematically misrepresents a key construct and produces misleading findings. For example, questions might classify researchers who use AI for drafting but retain final decision authority as having delegated their methodological decisions to AI. Severity: Potentially high. Such errors can distort conclusions about AI reliance, affect product or research decisions, and require redesign or data recollection. A survey draft can look polished and meet formatting requirements while still containing consequential measurement errors.

Detectability is mixed. Relatively easy to detect: Missing required questions, inconsistent response formats, and violations of survey-length constraints. Harder to detect: Subtle construct contamination, leading wording, and differences in how respondents interpret terms such as trust, reliance, and delegation. For example, an item asking whether someone "trusts AI enough to use its recommendations" may conflate perceived trustworthiness with actual reliance. Such problems may survive document review and only become visible through expert assessment, cognitive interviews, or later empirical analysis.

Before fielding: High reversibility. Researchers can revise wording, replace response options, add omitted constructs, and conduct further pretesting. After fielding: Much lower reversibility. If respondents interpreted a measure incorrectly, the original data may not support the intended construct, even if the survey is corrected later. For example, discovering after data collection that an item conflates AI assistance with decision delegation may require recollecting responses. Verification therefore needs to happen before consequential downstream use. Expert review effort must also be weighed against any drafting time saved by AI.

CASE 1 — TYPICAL Draft a survey measuring frequency of AI use, perceived usefulness, and verification behaviors, using a complete brief with clearly defined constructs. Tests whether AI translates well-specified objectives into relevant items and satisfies explicit requirements. CASE 2 — AMBIGUOUS Draft items measuring trust in AI-generated research outputs where the brief distinguishes perceived trustworthiness, willingness to rely, and actual verification behavior. Tests whether AI preserves related but distinct constructs and avoids treating trust, reliance, and behavior as interchangeable. CASE 3 — HIGH-CONSEQUENCE Draft items measuring willingness to delegate consequential methodological decisions to AI, including situations where researchers use AI but retain final decision authority. Tests whether AI avoids leading wording and misclassifying assistance as delegation. For each case, control the population, research context, construct definitions, constraints, required output, and allowed tools. Include multiple instances per case type where feasible. One successful draft cannot establish reliability across survey-design tasks.

Production conditions: Human-only, AI-only, and Human + AI, using equivalent task specifications and predefined quality criteria. Preserve the unedited AI outputs, prompting protocol, model/version information, and documented human interventions. At least two qualified quantitative researchers independently review blinded drafts before discussing provenance or disagreements. Use objective checks for explicit requirements and anchored expert ratings for methodological quality. Record evidence for each rating and distinguish critical defects from acceptable methodological variation. Pilot the rubric using deliberately flawed drafts, defensible alternatives, and boundary cases. Examine reviewer disagreements, failure severity, variation across task instances, and verification effort. Conclude only what the tested cases support. A well-designed evaluation does not guarantee acceptable AI performance, and a consistent rubric is not necessarily a valid rubric.

AI may: Generate candidate questions, suggest response formats, propose alternative wording, organize items by construct, and flag possible omissions. Provided that: The research brief specifies objectives, constructs, constraints, and requirements. AI outputs remain candidate drafts rather than validated instruments. Humans must own: Construct definitions, operationalization decisions, methodological appropriateness, final item selection, interpretation of expert feedback, and approval for pretesting or fielding. Review happens: A qualified researcher checks item-to-construct mappings, wording, response options, and potential measurement bias before approving the draft for pretesting. Cognitive pretesting is used where respondent interpretation remains uncertain. Escalate or stop when: AI introduces unsupported construct assumptions, omits required dimensions, repeatedly produces critical methodological defects, or cannot resolve ambiguity without inventing requirements. Human review must evaluate substantive measurement quality, not simply formatting or completeness.

Toward Delegate + Verify: Repeated evidence across representative and challenging cases that AI drafts meet predefined criteria, critical defects are rare or reliably detected, and verification is feasible and worthwhile. Toward Assist Only or Human-Led: Persistent critical defects, unreliable detection of measurement problems, or expert verification effort that eliminates the benefit of AI drafting. Any change should apply only to the task conditions supported by the evidence. Immediate next step: Create a standardized survey-design task packet, define three initial cases (typical, ambiguous, consequential), finalize criteria and critical-failure definitions before generating outputs, and pilot independent expert review. Use pilot findings to refine the evaluation before making an operational delegation decision.