Microsoft Build 2023 - Day Two
Shaping the Future of Work with AI - Panos Panay, Rajest Jha, Yina Arenas, Steven Bathiche, Pavan Davuluri, Wamwitha Love & Archana Saseetharan
Rajesh Jha who leads experiences and devices for Microsoft and talked on how we can work together to shape the future of work with AI. Copilot Stack is the AI architecture of the future and Microsoft are using this to bring that to various products. Graphical User Interfaces made computers accessible to more people around the world and AI is even more accessible and even more powerful and can use it to transform how people work in society and do their work.
Future of Work
What can 30,000 people tell us about work, the one thing they talked about was spending two days of their work week coordinating and communicating and most of them said they were not able to keep up with their work, most of the leaders of teams were saying they were not innovative enough= and for employees they were worried about how AI would impact their job but most wanted to delegate their work to AI.
Microsoft has many productivity tools to help people to get their work done such as Bing where people are researching and getting information with 100 million users and Windows which has a billion users and Teams which has 300 million users taking advantage of it and it has over 1,900 apps for Teams in the store to bring productivity in the flow of work.
Microsoft 365 Copilot can help shape work by putting users in the centre, it isn't auto pilot and is helping people achieve more and being grounded in their context and will see this in the design language that is used, and developers will be able to use the same design language and help create the most powerful productivity tool on the planet.
Microsoft 365 Copilot is not just about taking in the most powerful foundation models but is to ground this with data from the user's context using the Microsoft Graph and the user experience is brought to applications and workflows they are already using. There are two capabilities such as in-application assistant to help complete a task in hand, but there are also cross-app intelligence layer - which is like ChatGPT for the enterprise, this reasons over user data and applications and content and brings all these workflows from a myriad of context and applications and put the user in the centre and bring everything around them.
Microsoft 365 Copilot Cross-app capabilities - It is available where your users are such as Microsoft Word or Teams. Archana Saseetharan talked about how Copilot can be used in Microsoft Teams and showed how to find out when a monthly meeting will be and can do more than just fetch information but summarise their correspondence with the organiser of the meeting and can get specific details and information where needed and can ask Copilot to take actions on their behalf such as send an email with the information which can then be personalised or just copy and pasted and sent with Outlook, to help summarise and synthesise information and helps you work faster and more efficiently. Simple natural language interface to get the job done, that intelligently reasons over all your apps, user, and company data. Extensibility with plugins, this allows you to customise the user experience and have the Copilot to allow it to perform actions on the user's behalf and these plugins use an Open Standard to support many different AI experiences such as ChatGPT, Bing and more using an OpenAPI endpoint. This is not the only way to extend a Copilot with features such as Teams message extensions and Power platform connectors can work across Copilots as a new skill for the copilot - it is like the operating system for connectivity and business process where Developers build the extensibility. There will be 1,000s of plugins for Copilot which will also include the Teams message extensions and Power Platform connectors.
Archana Saseetharan showed how you can create plugins with Teams Toolkit for Visual Studio to develop, build and debug plugins. First you just need a RESTful API that follows the OpenAPI specifications, which can then be converted to a Plugin for Copilot, you provide an OpenAPI specification from the API and then select the methods of the API you want to expose in the plugin as well as a location and name for the project which will create a new project. A plugin has two key components which is a manifest which describes the configuration and metadata for the plugin and the adaptive cards that defines how users will interact with the application and these are created automatically in a new project. Adaptive Cards will be based upon the API and then can make changes and see a preview of these. You can debug a plugin by launching it to run with Copilot in Teams for testing and you can see the output of the Plugin and the behaviour from inside Microsoft Teams.
Using this Toolkit will make it easier for developers to create Plugins.
Microsoft Graph
Microsoft Graph is something that is added to every day by users, it partitioned by customer and by tenant and is permissioned by user, so whatever you have access to is what it can use, and this is the brain behind the Copilot's amazing reasoning capabilities. The data is not used to train the AI, it is used to ground it, this is treated the same way as a user's documents or content. Natural language requires an understanding of semantics with a Semantic Index for Copilot which is a tenant wide and user scoped vector database, identify semantics and similarities around content and users, enable fast semantic search across billions of vectors and you can extend this with graph connectors.
Yina Arenas talked about Graph Connectors to bring more value from your data into Microsoft 365 Copilot where you could use Copilot to gather information, then use this to invoke plugins in context such as a Jira ticket and then use this to get and update an asset, it would be more tedious to do this manually and would need to work across different interfaces but instead can do it without breaking the flow of their work. Copilot can traverse all the information in Microsoft Graph such as documents and emails but can also connect to Jira with a Microsoft Graph connector to get this information and can even provide citation and because it includes full semantic search so includes related information that may have been missed. Copilot can also use a message extension as a plugin in Microsoft Teams and can be used to add additional information. Copilot can also interact with an enterprise application such as to retrieve an asset which may use a message extension so Copilot could work with this and then work with context and invoking the correct actions that have are required. As a developer you can use graph connectors to get your data into Copilot and you can use plugins or message extensions and can provide insights from Microsoft Graph and reduce workload.
By using Microsoft Graph you get the full capabilities of the Semantic Graph.
In-app capabilities
The way this is extended is the same way that it is done across applications by brining the full power of the Microsoft 365 applications and unlock this with natural language. Wamwitha Love showed how an account manager who has an upcoming sales meeting that they need to prepare for so will use an integration with Viva sales to get their documents together and Copilot generates a summary for these, but then can ask for more details from Copilot and Viva Sales can be used to get more information and can update any documents as needed. Copilot can be used in Viva Sales itself to process a customer conversation in real time, such as they mention a competitor it can look up this information and can then responds with anything directly in the meeting. This can be followed up with need for a contract for the agreement using Copilot and it has the context it needs to create information and can use Plugins that have access to multiple trusted resources to perform specific legal adjustments to the document along with summarisation of any clauses.
Microsoft Syntex plugin for Microsoft 365 Copilot, which is content AI which allows you to structure and secure content for Copiloit, leverage your organisation knowledge, and find organise and manage content and be grounded in the user's context. It can automatically extract meta data and search and find relevant data with precision and generate new documents which can be reviewed and spotlighted with key details and can have eSignatures for approval.
Developer Success
Productivity - you can create new Plugins with Teams Toolkit and customise user experience with adaptive cards. Reach - need to make sure data privacy is maintained with tools to improve app compliance, speed up IT approval with Microsoft 365 certification, easy IT discovery and deployment of apps and auto install for approved apps across your organisation which is a win for developers, IT and users. Discovery - have to have end user get your application, may go for a recommendation from colleagues or IT with a consistent store experience across Microsoft 365 apps with seamless discovery in the flow of work with link unfurling and contextual access in Teams chats, channels and meetings and have added user experience to get the right application in the right context. Revenue - Microsoft have a commercial marketplace to transact with easy license management and tracking.
Call to action to developers
Go build Teams message extensions today, build graphic connectors to bring your data to Semantic Index for Copilot and sign up for the Visual Studio tooling preview.
Teams Live Share
Live app collaboration in meetings, co-create, co-watch and co-edit with no backend code required, in a hybrid workforce, you want your applications in a meeting you want them to be used by multiple people at the same time.
Microsoft Mesh
Building connection and building fun with immersive experiences with Avatars for Teams which offer present and expressiveness even with your camera off along with Immersive Spaces in Teams such as a town hall or for employee onboarding, make those hybrid meetings feel more personal and make people feel more included and be more engaged and feel more natural.
Future of Windows - Panos Panay
There is one more place you can interact with users, over a billion of them in Windows. Panos Panay talked about loving it when people are pumped about Windows and thanked all of those working on Windows and to those attending Microsoft Build in person and online. It is an incredible time to be a developer, especially developer on Windows. The possibilities across industries are endless, are moving so fast, and we can lose sight of what it actually means for us as people, what is this transformation for each of us.
Panos then asked a question, how many of you remember the first time or the early days of the internet? Panos remembered for them, they wanted to understand what it was, they wanted so bad to understand what people were talking about, there were these noises and a dial-up modem and used something called Gopher and then got into work and then got a connection and remember being on the internet and remember this more than the work they were doing. It was magic, the feeling was indescribable, even then in online infancy the opportunity felt so vast and impossible to comprehend and in those first few weeks they read everything they could. The level of knowledge at their fingertips was transformational, then came new form factors but that feeling of possibility they had never felt since until the advent of AI. The truth and opportunity of what is in front of us, it is part of getting to witness and participate in something that will change the very fabric of how we live our lives, and it is only just the beginning.
For the devs out there and the devs to be you might just be starting with AI and might not know where to start or may be an expert. No matter where you are on your journey it is about Windows being there for you. You don't go from the ground to the top, you need to take the steps along the way, and we are the wave makers, the creators, and the pioneers to change the world. Find out how to create engagement with Windows Copilot and plugins. Will find out how Windows is fundamentally the absolute best endpoint for AI and the home of every one of your high ambitions of AI models. If you are on Windows 11, are building it and using it, it is what will keep you ready as we pioneer though this opportunity together.
Panos wants developers to come with him on this journey, the first thing to understand, which he gets asked a lot, but not everyone knows how to get to AI, what is the first thing they do is bring AI front and centre and bring those plugins forward to a billion people and bring this to the talk bar, this makes every user a power user. Think about thirty years of history in a platform and make it easier, you might not even know a feature exists that can be surfaced, think about any setting in Windows, people think about things like needing to cast their screen to their TV, but they can just write it down and watch it happen. There is a nuance where the Copilot can talk to you and where a plugin can just wake everything up such as asking to listen to music and then can pick what you want, Windows can then recognise that you need to do something and will just offer it.
Panos talked about a Kubernetes manifest in Notepad and then can copy this into Copilot and can ask it to do something with it, you could get it to explain it, to save time. Think about this for third-party applications or anything that sits on your desktop today. You could generate a transcript from some audio to save some time, which is pretty powerful. The nuances of another demo are important which is how to move creators into their flow, it can just bring you a plugin and keep creating which you can then share, the steps of opening, closing and removing are gone. There is so much power there, when think about Plugins, Windows is your platform for high ambition AI apps and it is sitting right in front of us. Windows Copilot previews in June in Windows 11, get this in preview and will learn together, it will change your workflow and will deliver an AI experience and will evolve this together.
Windows 11 is goes beyond AI and it is such an important platform, it was announced two years ago and have more people getting on this version of Windows, the usage is up, but the growth extends first to developers, with more developers developing and making Windows a more critical tool to bring content to customers. Windows 11 has made a transformation for everyone to connect, play and work and there's no limit to what you can do. There are so many apps that have come to the Windows Store - these apps seem obvious such as Spotify, Messenger, Snapchat and What's App which were only mobile apps, the advent of AI is only making that a stronger, faster opportunity.
When Panos first took on Windows, the most passionate group that gave them feedback was developers, and this feedback was not gentle but was necessary and inspired them to make improvements to the UI. Today Microsoft want to bring the best Windows development experience for developers and called it Dev home.
Shilpa Ranganthan talked about needing to support Bash on Windows and supporting this with WSL and enabled a better experience with Windows Terminal and Powertoys is a top five supported project by developers. Microsoft like feedback on how to make Windows the best platform for developers, it takes too many clicks to get things started and that Windows should have more options for power users. Dev Home is a brand-new experience for developers to let you focus on what you are working on and easily install the tools and packages you need and connect this with GitHub all the way from File Explorer to Windows Terminal. This is powered by WinGet Config and can define this with a single file and apply this with a single click. Microsoft has worked hard on this feature for helping developers save time, Dev drive is a new way of dealing with large projects using the resilient file system and will help speed up build times over what Windows 11 can do today. Dev Home helps you by removing distractions, can get a dashboard of issues and CI/CD status on GitHub and track tasks and this extensible including support from the GTK for Xbox and developers can contribute, you can get it in preview today.
Shilpa Ranganthan also spoke about improvements to Windows 11 to improve efficiency and productivity includes native support for additional compression formats and can easily end a task from the taskbar, Windows Terminal will support tab tearouts and GitHub Copilot X will be integrated into Windows Terminal, bringing the power of AI into your command line and can get access to the new GitHub Copilot Chat. You can help engage customers with your app with Widgets where you can amplify your apps and services with support for third-party Widgets already added and will have more space for Widgets and will add additional layouts and have a mix of news feed and Widgets in the future and make widgets easier to discover. Retention in applications in new devices with a new improved restore experience which will include all of their store applications where they need them, and this is starting in preview. Microsoft Store has over a billion unique users and can bring any application type with outstanding revenue share. There are Microsoft Store ads which can help increase app installs so this will be expanded to 150 regions. Microsoft Store + AI which has an AI generated review summary to make it easier for fans to influence downloads and for developers to get an overview of customer thoughts. There will be a new AI hub in the Microsoft Store which will also educate customers on using AI and inspire them to use them in their everyday lives. Expend your customer reach with an AI-powered app.
Paven Davuluri showed how AI can work for every developer, if AI is new to you then Windows can get you started, and can help developers with Windows AI library, ONNX runtime. If you don't know where to get started there are AI models that can be integrated into applications easily. Microsoft are building a Windows AI Library with Windows Studio Effects and a large range of AI models coming soon. To get to the next level you need ONNX Runtime and Olive Toolchain to create experiences that are tailored for your customers.
Cassie Breviu spoke about ONNX which does all the heavy lifting on multiple platforms and allows you to target the cloud, these are the same technologies used in Microsoft 365 applications. Olive Toolchain brings all the optimisation steps and there are walkthroughs to help you get started and simplifies a workflow. You could use Olive Toolchain to optimise Stable Diffusion and reduce the time needed to create an image.
Paven Davuluri talked about the hybrid loop for the cloud and the client, where use the cloud to host large datasets although the concept is simple the implementation is complex but with ONNX Runtime this simplifies the creation of a hybrid loop.
Cassie Breviu then showed how the cloud model can process information but if want to run locally can use an Olive Optimised small version of a model for local execution instead to use the ONNX runtime across different platforms.
Paven Davuluri talked about how you can accelerate the development of AI, you could have GPUs for acceleration of models with over 200 million users having them. NPUs are powerful ways to accelerate AI workloads along with dedicated support for ONNX runtimes on hardware and a lot more NPU devices coming to the market such as those used on the Windows Dev Kit 2023 which was announced at Microsoft Build 2022. With Windows, AI enabled silicon, ONNX runtime and Olive Toolchain are making every developer an AI developer.
Steve Bathiche spoke about their thoughts, AI is bringing unprecedented change and makes them feel like an intern again, need to think about the impact on our apps. Between Windows, Microsoft 365 and Azure ML are getting the most powerful tools to make AI your own. Getting the tooling to fine tune models with just one click and making a model like CPT-3 your own that was so difficult a year ago is now so easy. Utilise the user's context, personalise every interaction and every prompt, this contextualisation allows you to go broad and deep. AI is the new interaction technology, mouse and keyboard was introduced fifty years ago, but for the next fifty years interaction will be completely transformed be less programmatic and more piloted. Three new structures have emerged with AI, is it beside, inside or outside your app? Beside- Add copilots to your application and keeps the original application architecture definition and delivers immensely capable tools and experiences that were not possible before. Inside - create apps with AI at the centre of the experience to completely redefine the experience and take pro-level experiences and turn then into one click, fewer toolbars and menus. Outside - AI goes beyond context and is executed globally and help orchestrate across apps and tasks acting as an agent and go across workflows and devices. The Windows shell itself is an orchestrator, with AI you start to see glimpses of the capabilities of a Windows Copilot - a Copilot of Copilots and with a plugin model already have outside structures you can create today. Solutions that are provided today allows you to develop for one or all of these, may feel like you are learning again but Microsoft are alongside you, each of the solutions build upon the next, and help reinvent your interaction model with beside, inside and outside and make work more powerful and intuitive to help reach and empower more people.
Developer joy with Scott Hanselman and friends - Scott Hanselman & Kayla Cinnamon
Scott Hanselman and Kayla Cinnamon are developers and are excited to announce Dev Home, which is the new home for developers which is fully open source and extensible and any dev on Windows can find something helpful, it also has a fully customisable dashboard. Dev Drives is coming to Windows 11 which uses REFS which is a Workstation quality file system.
WinGet Configuration lets you get into a desired state to get a setup done instead of a Batch File and will support other desired state configuration providers. You may just have got a Word Document which you need to follow but instead you can just use WinGet Configuration. Scott opened up a brand-new Surface Laptop Studio and then used the OOBE to get through this. Onboarding is a hassle, you want to get started as soon as possible can then have Dev Drive, Dev OOBE, WinGet and Dev Home to remove developer toil and once started you want that developer inner loop to be easier. Microsoft want to centralise this experience and make it easier for developer when they code, build, test and deploy - you want this loop to be as quick and tight as possible, where things like Copilot will make it more fun and performance matters at this point. When Windows 11 is being installed there is a “Developer” option which will install Dev Home, but it could also do other things and how to get to the inner loop and outer loop faster. Then there will be your favourite features you want to install and there is a Dev Home extension for GitHub. Once Dev Home is launched you can go to the Machine configuration page and can use a YAML file to install the things you need such as your development environment such as Visual Studio. A lot of the ideas that are being implemented came from the community and were able to replicate the open-source process of Windows Terminal to Dev Home. You can customise the Dashboard and see system widgets such as CPU, GPU and assigned tasks from GitHub. You can create a Dev Drive from the System where you can create a new VHD which will dynamically resize, you can also have something on an existing disk, you can have as many drives as you want or share them as needed with performance improvements when building and Windows Insiders can try it now and check for improved performance or reduced performance. If you are a certain kind of developer Dev Home could set up things for a particular type of developer, although this is not supported yet.
Windows Terminal can be the default terminal in Windows 10 as well as Windows 11 and supports tab tear off and you can move tabs between windows. When building ASP.NET itself this could be quite difficult so can use the WinGet Configure to set up a machine with all the things that are needed, you can even customise Windows Terminal to add retro terminal effects using a shader to make this happen which can even include scanline effects, it is just awesome - being able to do stuff!
New developer experiences in Windows - Kayla Cinnamon & Sharla Soennichsen
Microsoft are listening and keeping tabs on what people are posting online along with hundreds of customer interviews and engaging with MVPs. Developer lifecycle for onboarding is purchase, device setup and environment setup then inner loop of code, compile, debug and commit and the outer loop is where the CI / CD pipeline is.
Dev Home
What exactly does Dev Home include - it is a centralised location for your dev needs, integrated machine setup flow, open source and extensible and available today on Microsoft Store and GitHub. It comes with a GitHub Extension installed by default so you can manage your GitHub Repositories and you can build your own extensions.
When you first launch Dev Home you get the introducing Dev Home page and then you can connect to GitHub and then log in then can go the Dashboard with Widgets that are built with the same platform with a Widget Host so those Widgets will work there and include system performance widgets for GPU, Memory, Network and CPU and have SSH keychain along with GitHub Widgets for Review requested, Issues, Pull Requests, Mentions and Assigned to Me. You can monitor what is most important to you and can install other extensions from the Microsoft Store and there will be an Azure Dev Ops extension.
Dev Drive
Dev Drive is new trusted drive with improved performance with 30% performance improvement with Resilient File System (ReFS) and performance mode for Microsoft Defender Antivirus that reduces the impact to your workflow and can have filter manage control for faster performance in your Dev Drives. Brings the performance, security and control that can only be delivered by Windows. You can create them with the size you need to accommodate your files and select the drive letter you want to and the minimum size is 50GB as ReFS is more metadata intensive and will be bit locker encrypted automatically if enabled. When signed into GitHub can see all the repositories and easily clone these to the Dev Drive automatically, or these could be created as a VHD during the setup process if one doesn't exist already. In Dev Home you can install any applications you want such as Visual Studio Code.
WinGet Configuration
WinGet is the Windows package manage command line tool and WinGet Configuration is a repeatable reliable mechanism to get your machine to a code state as quickly as possible and can handle the setup and requirements of a development machine, this includes setup of languages, tools and settings needed for a project. You can get a Configuration File which is a YAML file, which are also used by WinGet manifests, and it can check version and make sure have Developer Mode turned on and install Visual Studio with the relevant workloads.
Updates
Extension to Windows Terminal is being developed to be able to use the chat with GitHub Copilot and it is also in WinDbg where can type in natural language requests for things to do there powered by AI. Default terminal support is coming to Windows 10 to use Windows Terminal and there are new features including tab tear out, portable mode and environment variables in settings. Windows Subsystem for Linux can now run systemd which has basic building blocks to interact with a Linux system, it is also delivered undocked from the operating system and updates will be pulled from the store and no longer attached to the Windows version. Updates to PowerToys include PowerToys Peek, Paste as plain text, registry file previewer, Hosts file editor and Mouse without Borders and has been made open-source to use one mouse and keyboard on two separate PCs. There is also a new command not found WinGet suggestion and allow it to install what is needed for that command and continue in your workflow in one place as a PowerToy. Additional file compression support will be using the open-source lib archive library to support 7zip, tar and rar files and will add support for creating those files next year. The Out Of the Box Experience (OOBE) has added representation for developers and has a Development checkbox which will automatically add Dev Home and want feedback about what should be installed when this is checked. Windows Settings have been updated with the ability to hide the clock in the system tray, there is a task bar setting to never combine the task bar buttons to make it easier to see what is open. End Task in Taskbar is now a setting so when you go to the menu in the taskbar you can End Task.
What's next?
Dev Home - Azure DevOps Support, Multiple dashboard support - will be working on multiple things so support for this will be added, deeper integration with repositories - how can link things together in once single flow and gaming development integration - Windows and team Xbox are making it easier by bringing the Xbox development kit to Dev Home with the public GDK being available on WinGet.