VOOZH about

URL: https://dev.to/ramunarasinga-11/web-worker-for-background-json-parsing-in-langfuse-codebase-3f32

⇱ Web Worker for background JSON parsing in Langfuse codebase. - DEV Community


In this article, we review how Langfuse parses JSON using web workers in the background. You will learn:

  1. What is Web Worker?

  2. json-parser.worker.ts

  3. useParsedTrace.ts

👁 Image

What is Web Worker?

Web Workers makes it possible to run a script operation in a background thread separate from the main execution thread of a web application. The advantage of this is that laborious processing can be performed in a separate thread, allowing the main (usually the UI) thread to run without being blocked/slowed down.

A worker is an object created using a constructor (e.g., Worker()) that runs a named JavaScript file — this file contains the code that will run in the worker thread.

Worker

The Worker interface of the Web Workers API represents a background task that can be created via script, which can send messages back to its creator.

Creating a worker is done by calling the Worker("path/to/worker/script") constructor.

Workers may themselves spawn new workers, as long as those workers are hosted at the same origin as the parent page.

Learn more about Web Worker.

json-parser.worker.ts

In the json-parser.worker.ts in Langfuse codebase, you will find this comment.

/**
 * Web Worker for background JSON parsing
*
* Parses large JSON data off the main thread using the iterative
* deepParseJsonIterative function to prevent UI blocking.
*/
import { deepParseJsonIterative } from "@langfuse/shared";
export interface ParseRequest {
id: string;
input: unknown;
output: unknown;
 metadata: unknown;
}
export interface ParseResponse {
 id: string;
 parsedInput: unknown;
 parsedOutput: unknown;
 parsedMetadata: unknown;
 parseTime: number;
}
self.onmessage = function (e: MessageEvent<ParseRequest>) {
 const { id, input, output, metadata } = e.data;

 const startTime = performance.now();
try {
 // Parse with high limits since we're off the main thread
 const parsedInput = deepParseJsonIterative(input, {
 maxDepth: Infinity,
 maxSize: 10_000_000, // 10MB
 });
const parsedOutput = deepParseJsonIterative(output, {
 maxDepth: Infinity,
 maxSize: 10_000_000,
});

const parsedMetadata = deepParseJsonIterative(metadata, {
 maxDepth: Infinity,
 maxSize: 10_000_000,
});
 const elapsed = performance.now() - startTime;

 const response: ParseResponse = {
 id,
 parsedInput,
 parsedOutput,
 parsedMetadata,
 parseTime: elapsed,
 };
 self.postMessage(response);
 } catch (error) {
 console.error("[json-parser.worker] Parse error:", error);
 // Send back unparsed data on error
 self.postMessage({
 id,
 parsedInput: input,
 parsedOutput: output,
 parsedMetadata: metadata,
 parseTime: performance.now() - startTime,
 error: error instanceof Error ? error.message : String(error),
 });
 }
};

Langfuse uses the web workers to parse large JSON. This deepParseJsonIterative function is imported as shown below:

import { deepParseJsonIterative } from "@langfuse/shared";

useParsedTrace.ts

When I searched for json-parse.worker. I found two instances:

👁 Image

In the useParsedTrace.ts, the following code is defined to create a worker

function getOrCreateWorker(): Worker | null {
 if (typeof window === "undefined" || !window.Worker) {
 return null; // SSR or no Worker support
 }

 if (!workerInstance) {
 try {
 // Next.js will bundle this as a separate chunk
 workerInstance = new Worker(
 new URL("@/src/workers/json-parser.worker.ts", import.meta.url),
 );
 workerInstance.onmessage = (e: MessageEvent<ParseResponse>) => {
 const callback = pendingCallbacks.get(e.data.id);
 if (callback) {
 callback(e.data);
 pendingCallbacks.delete(e.data.id);
 }
 };
 workerInstance.onerror = (error) => {
 console.error("[useParsedTrace] Worker error:", error);
 };
 } catch (error) {
 console.error("[useParsedTrace] Failed to create worker:", error);
 return null;
 }
 }

You will find that this is used in components/sessions/index.tsx

About me:

Hey, my name is Ramu Narasinga. Email: ramu.narasinga@gmail.com

Tired of AI slop?

I spent 3+ years studying OSS codebases and wrote 350+ articles on what makes them production-grade. I built an open source tool that reviews your PR against your existing codebase patterns.

Your codebase. Your patterns. Enforced.

Get started for free — thinkthroo.com

References

  1. langfuse/web/src/workers/json-parser.worker.ts

  2. github.com/search?q=repo%3Alangfuse%2Flangfuse%20json-parser.worker&type=code

  3. langfuse/web/src/hooks/useParsedTrace.ts#L28

  4. langfuse/blob/main/packages/shared/src/utils/json.ts#L61