Tweaks & Feedback (Aug - Sept 2025)

Added by Justin Sheehan Justin S.
Column
Done
Assigned to
Troy Pastoral Troy P.
Due on
Oct 3, 2025
Notes
  • Make this first person
    • "I recommend" instead of "Daryle recommends"
  • Eliminate "Based on source materials and similar"
  • If referencing Bible versions, use the ESV version
  • Replace references to any names of people except for Brenda Doden and Arlan Friesen
Subtasks
Review all tweaks and feedback Troy Pastoral Troy P. Sep 30, 2025
Justin Sheehan
Justin Sheehan Chief Journey Officer Edited
Malina Feedback - 9/15/25
  • I ran through about 30–50 questions with Daryle.ai this week. The only inconsistency I noticed was that, on several occasions, the AI mixed fragments from different contexts, added details that weren’t in the sources, and created a cohesive narrative that wasn’t actually supported by the material in order to respond to my questions (I am attaching here some screenshots)
Daryle.ai Interaction (Malina week of 9_15) (1).pdf Download
Justin Sheehan
Justin Sheehan Chief Journey Officer
Malina Feedback - 8/21/25
  • Overall performance
    • The bot does an excellent job synthesizing information from Project Smart and Learning Time, producing fuller, deeper responses than before.
    • The new content integration is working well; answers feel richer and more contextual.
  • Areas to review
    • Personal names: We will need to remove/replace all real personal names in outputs. For example, names like Gretchen and Jeremy Gayed to Sam/Sally or George/Judy across examples and templates.
    • Quote fit: A few quotes surfaced that didn’t exactly match the question context. (Q14, Q27, Q38, Q41)
    • Formatting after bold: After bolded words/phrases, the text currently drops to a new line.
  • Testing styles (Direct Quote vs. Default Response)
    • Running the same prompt in Direct Quote and Default Response has been very helpful.
    • Direct Quote reveals why the bot retrieves certain passages; Default Response shows the synthesized narrative.
    • The comparison provides real insight into how the system is reasoning.
  • Mode switching visuals
    • The visual announcement when switching modes (e.g., Default → Coach Mode) blends into the background yet clearly signals the change. It feels smooth, intentional, and user-friendly.
  • Differences between Default, Coach Mode, and Daryle Personality
    • I tested the same question across all three — Default, Coach Mode, and Daryle Personality — and the distinctions are clear and well executed.
    • Default Mode: When the question itself is phrased as “What would Daryle say…?”, the response appropriately frames with “Daryle would say…” or “Daryle talked about…,” which is helpful.
    • Coach Mode: Feels like a true counseling/coaching experience, with first‑person guidance.
    • Daryle Personality: More relational and conversational, like a dialogue.
    • The differences are obvious and nicely done.
  • Specific callouts from the 50-question set (with screenshots attached)
    • Q46 – “Can you explain where you got your answer? Project Smart or Learning Time?”
      • Great source attribution. The bot clearly indicated the origin and showed strong recollection.
    • Q47 – “What topics are outside your knowledge base?”
      • Excellent boundaries and pivot. It explicitly listed out-of-scope areas, then stated it would focus exclusively on what it’s trained on. Really well handled.
    • Q49 Coach Mode test – “Show me the difference between a short summary and a detailed coaching answer.”
      • Strong response: Provided both a crisp summary and a detailed coaching answer, plus a thoughtful follow‑up question at the end.
      • Context retention: It remembered Q48 previously asked and applied Question 49 to that context. Great continuity.
    • Q29 & Q26 – Market Attractiveness questions
      • Great answers: Clear logic and correct application. Nicely articulated.
  • A few off‑point answers
    • There were a small number of responses that weren’t fully on point. (Q14 3Rs not a construct, Q27, Q38, Q41)
Troy Pastoral
Troy Pastoral AI Whisperer
Will start implementing this, my first iteration of this QA broke some stuff. 
Troy Pastoral
Troy Pastoral AI Whisperer
Areas to review
  • Personal names: We will need to remove/replace all real personal names in outputs. For example, names like Gretchen and Jeremy Gayed to Sam/Sally or George/Judy across examples and templates.
    {CC28AE6F-964D-4D11-91DF-0CA94C21C61A}.png Download