Microsoft Semantic Kernel for AI Dev: A Chat with John Maeda

Microsoft recently open sourced a product called Semantic Kernel, a lightweight software development kit (SDK) allowing developers to integrate AI technology into their applications. To find out the thinking behind Semantic Kernel (which Microsoft abbreviates as SK) and how it might be used by developers, I conducted an email interview with the project leader, John Maeda — who is also VP of Design and Artificial Intelligence at Microsoft.

In his announcement post, Maeda wrote that SK “lets you mix conventional programming languages, like C# and Python, with the latest in Large Language Model (LLM) AI “prompts” with prompt templating, chaining, and planning capabilities.”

The Art of Prompt Engineering

I began by asking what kind of skill sets developers will need to learn to become good at these “prompts”?

“Traditionally, computer science education has been about achieving structured outputs from well-structured syntax,” he replied, “but actually the more flexible mindset of data scientists or even creative artists can be assistive in order to navigate this new world.”

He said that a key skill is “choosing the right words, phrases, symbols, and formats that can guide the model to generate high-quality and relevant texts.” But in addition, developers will need to become proficient at “prompt chaining” (using generated texts as the basis for the next prompts), “prompt tuning” (adapting and optimizing the prompts for specific tasks or domains), and “prompt testing” (which is exactly that: testing the effectiveness of the results).

Sematic Kernel is basically a low-code tool that lets developers, as Maeda put it, “craft complex chains of LLM AI prompts that are both configurable and testable.”

So is “prompt engineering” just an evolution of what a developer will have to do, I asked, or will it be an opportunity for other professions to re-skill (I mentioned psychologists and English teachers)?

“The software world has been dominated by people who can literally speak machine,” he replied. “So, it’s an interesting turn of events that this new kind of programming is much closer to natural language. If your goal is to produce writing as an output, then there’s certainly room for many language-proficient English teachers to have an impact. That said, to be a productive ‘prompt engineer’ still requires you to have the ability to think like an engineer. There’s a reason why the engineering field emerged as a discipline; it’s always attracted those who love to build machinery. In the future, we can count on prompts that are engineered by developers to have qualities we both need and want — like reliability and efficiency. That won’t change. The difference is that developers will be able to pair up with AIs to create even more reliable and efficient systems than ever before.”

SK Concepts

There are several new concepts in this product, the primary ones being “Skills,” “Memories” and “Connectors.” In the documentation, a “skill” is defined as “a domain of expertise made available to the kernel as a single function, or as a group of functions related to the skill.” Memories provide further context, while connectors allow you to connect to external data and actions.

According to Maeda, the approach here is “goal-oriented AI.” He described Skills as “the core building blocks in SK” and noted that they can be both simple (“Please summarize this piece of text for me”) or complex (“Please summarize everything I need to know for today and build a plan for how I need to get done what I need to accomplish”).

“Memory increases the capability of a Skill tremendously by allowing you to tie the Skill’s capability to your historical data that can be persisted and accessed at any time,” he said, while Connectors “are customizable resources that enable external data access.” He added that connectors help address one of the primary criticisms of LLMs — that they are pre-trained, and so “essentially frozen in time.”

Let’s Speak Machine

John Maeda is well known not just as a technologist, but as a digital artist too. He’s written several books on the intersection of computing and design, such as “Design By Numbers” and “Creative Code.” His most recent book, “How to Speak Machine: Laws of Design for a Digital Age,” was published in 2019 and seems especially relevant to Semantic Kernel.

I asked whether SK’s concepts are a method to enable humans to “speak machine” more efficiently.

“As someone who’s luckily been tasked to explain many of the concepts of Semantic Kernel, and in turn enable legacy industries to adopt what’s a seminal shift in how we think of building AI-based features, I definitely think so,” he replied. “The first phase will be developers adopting this kind of mindset when creating software systems. They’ll need to gradually adopt the thought process to be gained from what we’re calling Schillace Laws — a set of best practices when coding across native code and semantic AI, as introduced by Microsoft Deputy CTO Sam Schillace. Central to Sam’s thesis is that we’ll need to speak both machine (determinism) and human (non-determinism) within engineered systems of the future. This is in line with the theme of the last chapter of ‘How To Speak Machine,’ which is that people who speak machine will need to speak better human.”

Of course, developers already have a leg up on the rest of us in terms of “speaking machine.” In the SK announcement post, Python and C# were mentioned as supported programming languages. But what about the web’s most popular language, JavaScript?

Maeda said that in addition to C# and Python, “we are also looking carefully at TypeScript and will add other language support based on feedback.”

I asked what have been the early use cases for SK among developers. He prefaced his response by saying that Microsoft has been hosting hackathons with both startups and enterprises.

“It turns out there have been many longstanding ‘mid-hanging’ fruit problems,” he said, “like an education app needing to have a larger inventory of appropriate quiz questions, or a customer support app with manually tagged cases that suffer from never getting properly tagged, or a process monitoring app that should email stakeholders when important edge cases trigger that are impossible to express in Boolean logic. There are a bunch of ‘too computationally cumbersome to code’ problems out there that when SK lands in a developer’s toolbox, developers can immediately recognize how to demonstrate a working solution with less than an hour of coding.”

The Shift from Syntax to Semantics

Lastly, and speaking of semantics, I asked why did Maeda and his team use the word “kernel” in the product name? It implies something very specific in computing — the part that has “complete control over everything in the system,” as Wikipedia defines it. Pretty scary when talking about AI, I noted over email (with a smiley face emoji, to indicate that I was half-joking).

“It’s a tip of the hat to that all-time enabler of computational productivity, the UNIX kernel,” he replied. “For those of your readers who remember when the UNIX kernel emerged, I think we all were a little confused by commands comprised of two characters ‘ls’, ‘cd’, ‘ps’ etc. But the big ‘a-ha’ was when we piped commands with the ‘|’ symbol and suddenly the light came on. The UNIX kernel’s simplicity as a landmark user experience for developers has been the north star for SK during its evolution. And we definitely don’t feel we have it right yet. That’s why we released it as open source. So that we can learn in the open as a community, and hopefully together build the right user experience for developers who are excited as we are by this new shift from syntax to semantics.”