VOOZH about

URL: https://phabricator.services.mozilla.com/D303489

⇱ ⚙ D303489 Bug 2037171 - Load chat prompts from v2 records and add get_skill tool r?bjohns,tetchart


Bug 2037171 - Load chat prompts from v2 records and add get_skill tool r?bjohns,tetchart
Needs ReviewPublic

Authored by mzhang on Fri, May 29, 8:03 PM.
Tags
None
Referenced Files
F72883143: D303489.1781746971.diff
Wed, Jun 17, 1:42 AM
Unknown Object (File)
Tue, Jun 16, 5:45 AM
Unknown Object (File)
Mon, Jun 15, 6:13 PM
Unknown Object (File)
Mon, Jun 15, 2:21 PM
Unknown Object (File)
Mon, Jun 15, 12:15 PM
Unknown Object (File)
Mon, Jun 15, 12:13 PM
Unknown Object (File)
Mon, Jun 15, 12:13 PM
Unknown Object (File)
Mon, Jun 8, 9:20 AM

Details

Reviewers
bjohns
tetchart
simonf
Bugzilla Bug ID
2037171
Summary

Teach PromptLoader to assemble the chat system prompt from modular v2 records
(identity, model-details, style, skills, trust-and-safety, response-rules,
browser-context) when the new pref points
at a JSON dump of the ai-window-prompts Remote Settings collection. Falls back
fragment-by-fragment to legacy v1 records when v2 is absent, so empty-dump
behavior is byte-identical to the prior patch.

New PromptLoader exports: buildChatSystemPrompt, buildBrowserContextPrompt,
getSkillPrompt. A new get_skill tool is surfaced when the v2 dump pref is set
and filtered out by Chat.filterFeatureGatedTools when empty. The chat-system
v2 assembly hooks into composeSystemPrompt's CHAT branch (post-Tyler refactor
in D302389), so v2 records replace the legacy single-record path while
preserving the {body, version} return shape. ChatConversation's
injectRealTimeContext collapses to a thin wrapper around
buildBrowserContextPrompt + a #lastBrowserContext dedupe guard so repeated
turns don't re-inject identical context.

Diff Detail

Repository
rFIREFOXAUTOLAND firefox-autoland
Branch
HEAD

Unit TestsBroken

TimeTest
0 mscode-review::general
WARNING: A generic error occurred in the code review bot.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Gijs removed a reviewer: Gijs. Gijs added 2 blocking reviewer(s): tetchart, bjohns.Tue, Jun 2, 9:02 AM
Comment Actions

I don't have a lot of context here and the fact that frontend is in the review list at all is probably a bug. But I've taken a look and added questions inline. Tyler and/or Bryan should probably be reviewing this more closely.

browser/components/aiwindow/ui/modules/ChatConversation.sys.mjs
752

Why do we only pass along if the signature is different? What is the "signature" reflecting, really? What breaks if we always update the value - is there a performance difference?

Note that it's probably a bug how much of this code lives in - almost all of it is directly talking to model code, it's confusing that the frontend needs to care about the specifics of how the prompt is generated.

754

This used to assign the whole object. Why are we now assigning only the prompt, but keeping the name?

806

Why can't tests stub the function directly? Having a function wrapper that directly calls some other function but does nothing else is a bit smelly.

816–820

The extended commit message suggests that all this is behind a pref, but AFAICT this now unconditionally loads v2 prompts. Is the commit message wrong, or am I missing some part of how this is supposed to work, or...?

tetchart requested changes to this revision.Tue, Jun 2, 7:42 PM
tetchart added inline comments.
browser/components/aiwindow/models/Chat.sys.mjs
138–140

restore comment

browser/components/aiwindow/models/PromptLoader.sys.mjs
30–31

Consider renaming to be more obvious. PROMPT_ARCHITECTURE_VERSION

69–92

Doesn't remote settings already have a local cache? Why are we doing another local cache here? Logic should be check remote settings, if prompt v2 not there, trigger a sync, check again

99–105

More caching stuff that I think is not relevant?

107–109

This feels like evals suite convention where it is openai/gpt-oss-120b, I don't think it's necessary here in MC

111–113

Why is this necessary? What model has "." in it?

124–146

This file feels like it has both prompt loading specific behavior and small helpers. Consider either organizing sections under comment blocks to help understand what is happening or even splitting file into relevant responsibilities

197–207

nit: Doesn't feel like it needs to be function

237–239

Are we worried some prompts may not be found? I think more explicit error handling here would be helpful. Why might this happen? Maybe log an error message?

240

Why is tableInstructions passed to this function, but format skill list is done here? Feels like there should be 1 place where we are collecting params to format?

249–251

Are we handling this error elsewhere? This name should be generated by the LLM so error handling is fine, but you are already checking later if the skill exists so not sure what this is really defending against? Either the name is valid or it's not and findSkill should tell us that?

273–274

It excludes those because our design choice in the prompt excludes them. Comment is misleading. this should include anything relevant to the prompt.

292–296

Why are we doing a backup with legacyFeatures?

300–306

Date isn't part of browser context anymore right? Because it's part of system prompt?

318–322

Where are these being set? If you delete, will they be reset at some point?

327

Do we need to sanitize url as well? Is this matching existing functionality?

416

If only relevant with chat system prompt, move into buildChatSystemPrompt

423–425

Is this to support v1? Or what is this?

430–437

nit: Is this common style in Firefox? I don't love having a special function that just makes test imports easier. I'd prefer to just import them in tests and not add extra code here

browser/components/aiwindow/models/tests/xpcshell/test_PromptLoader_buildBrowserContextPrompt.js
11–20

I don't think we are doing date in the browser context anymore?

browser/components/aiwindow/ui/modules/ChatConversation.sys.mjs
816–820

Reviewing these patches, I'm wondering what the benefit of supporting v2 prompts behind a pref is? v2 is about a new architecture to load prompt in a more scalable way for the future, I'm not sure I see a purpose to allowing us to go backwards to v1 prompts as they will not be supporting (tuned, made for new models, etc.) and here it is just adding complexity to try to support both.

I would consider this a breaking change to allow only v2 prompts going forward.

This revision now requires changes to proceed.Tue, Jun 2, 7:42 PM
Gijs added inline comments.
browser/components/aiwindow/ui/modules/ChatConversation.sys.mjs
816–820

I have very little context so take with a grain of salt. But, based on what you're saying, I think the only benefit is if we expect that there are situations where we want to have an emergency switch to go back to the previous version if we find breakage with v2, if we want to experiment (A/B test) between the two, or if the patch (stack) is somehow incomplete in terms of feature support, and we want to keep the v2 stuff behind a pref until it reaches parity. If none of that is the case then I agree that there probably is no real reason to keep support for the v1 version.

mzhang updated this revision to Diff 1298166.
mzhang edited the summary of this revision. (Show Details)
Comment Actions

Round 2 review applied. Replies inline.

browser/components/aiwindow/models/Chat.sys.mjs
162

Done.

162

Done.

browser/components/aiwindow/models/PromptLoader.sys.mjs
28

Done.

67

Done. loadV2Records calls getRemoteClient().get() directly now.

97

Done.

105

Done. Dropped the model normalization. Records hold the model identifier verbatim now.

109

Done, dropped together with the slash split. Records use the model identifier as is.

122

Done. Added section header comments organizing the file into records lookup, prompt assembly, feature config, and the public entry point.

195

Done. Inlined the date and template value helpers into buildChatSystemPrompt.

235

Left as silent skip since some modules may legitimately be absent. Happy to add a console.warn if you want a louder signal when an expected module is missing.

238

Done. tableInstructions, skill list, and the rest of the template values are now collected together inside buildChatSystemPrompt.

247

Kept the regex as a defensive guard since the LLM generates the name. Added a brief comment explaining why both checks are there.

271

Not sure which comment after the rebase. Most of the helper level comments got rewritten this round; let me know if a specific one is still off.

290

Done. Removed the legacy fallback in buildBrowserContextPrompt entirely.

298

Done. Date fragment removed from buildBrowserContextPrompt; date lives only in the chat browser context system prompt module.

316

The mapping is freshly built per call by constructRealTimeInfoInjectionMessage, so the delete only affects that one render. The next call gets a fresh mapping.

325

Matching existing functionality. The legacy code didn't sanitize the URL either, only the label. Can do it as a separate change if you want.

474

Done. Template values and skill list are inline in buildChatSystemPrompt now.

481

It was a v1 fallback. Removed entirely now.

497

Done. Removed PromptLoaderTestExports. Tests import the public functions directly and stub the RS client via head.js.

browser/components/aiwindow/models/tests/xpcshell/test_PromptLoader_buildBrowserContextPrompt.js
10

Done. Test rewritten without the date assertions.

browser/components/aiwindow/ui/modules/ChatConversation.sys.mjs
735

Removed the dedupe entirely. injectRealTimeContext always writes the rendered prompt now.

737

Not sure I follow. Pre and post patch both assign the rendered string onto userMessage.content.userContext.realTimeContext. Happy to look closer if you can point at the line.

833

Done. Dropped the seam; the one test now stubs the RS client and passes a getRealTimeMapping override.

839

Done. Pref is gone; v2 is the only chat path now, with a soft fallback to the legacy prompts field if module records are missing.

Revision Contents

PathSize
browser/
components/
aiwindow/
models/
7 lines
279 lines
27 lines
tests/
xpcshell/
15 lines
117 lines
235 lines
58 lines
49 lines
8 lines
ui/
modules/
59 lines
test/
xpcshell/
38 lines
CommitTreeParentsAuthorSummaryDate
f96d8abeea354e37a62af2cb711b34f3c5b9MoHan Zhang
Bug 2037171 - Load chat prompts from v2 records and add get_skill tool r?bjohns… (Show More…)
Thu, Jun 11, 10:57 PM

Diff 1301768

browser/components/aiwindow/models/Chat.sys.mjs

Loading...

browser/components/aiwindow/models/PromptLoader.sys.mjs

Loading...

browser/components/aiwindow/models/Tools.sys.mjs

Loading...

browser/components/aiwindow/models/tests/xpcshell/head.js

Loading...

browser/components/aiwindow/models/tests/xpcshell/test_PromptLoader_buildBrowserContextPrompt.js

Loading...

browser/components/aiwindow/models/tests/xpcshell/test_PromptLoader_buildChatSystemPrompt.js

Loading...

browser/components/aiwindow/models/tests/xpcshell/test_PromptLoader_getSkillPrompt.js

Loading...

browser/components/aiwindow/models/tests/xpcshell/test_Tools_GetSkill.js

Loading...

browser/components/aiwindow/models/tests/xpcshell/xpcshell.toml

Loading...

browser/components/aiwindow/ui/modules/ChatConversation.sys.mjs

Loading...

browser/components/aiwindow/ui/test/xpcshell/test_ChatConversation.js

Loading...