Introducing Copilot in Microsoft Dynamics 365 Guides, bringing generative AI in mixed reality to frontline workers

At Microsoft Ignite 2023, we are announcing Copilot in Microsoft Dynamics 365 Guides, which combines the power of generative AI with mixed reality to help frontline workers complete complex tasks and resolve issues faster with less disruption to the flow of work.

Copilot in Dynamics 365 Guides assists workers in industrial settings who deal with complex equipment and processes. New technicians need to ramp up quickly while learning how to operate and service that equipment. When equipment stops functioning, workers must act quickly to diagnose the problem, implement a fix, and then document their work. Instead of workers needing to search through volumes of digital documents or paper manuals, Copilot enables workers to ask for relevant information using natural language and human gestures, such as looking and pointing. Copilot uses generative AI to search for information from technical documentation, service records, training content, and any other data sources that customers curate. Copilot then summarizes the information to provide interactive guidance through content and holograms overlaid on the equipment in operation or maintenance.

Mixed reality together with AI presents a human-first interface that promises to transform real-world operations. Now, Copilot can assist industrial workers in real-time with speech and holograms.

Copilot in Dynamics 365 Guides capabilities

Here are the capabilities we plan to unlock with Copilot in Dynamics 365 Guides:

Point and Ask Guidance

Humans have always learned from and interacted with their environment in a hands-on manner, asking questions of others as they work. Copilot in Dynamics 365 Guides builds on that natural way of interacting to help workers better understand the world around them. Workers can point to a component on an asset and ask Copilot questions, such as “What is the torque limit for these bolts?” New workers can ask general questions, such as “What is the function of this component?” without needing to interrupt a more experienced coworker. Copilot recognizes what the user is pointing at and provides answers. 3D models and spatial maps provide the foundation for the technology to identify what the worker is looking at and then project a hologram to show instructions. Workers can ask more complex questions, such as “Which direction does this part fit?” or “Show me where the sensor module is on this machine,” and Copilot will display holograms to show where the part belongs.

Spatial Content Generation

Copilot in Dynamics 365 Guides accelerates the out-of-the-box value of mixed reality by significantly reducing the upfront investment needed to create spatial content. Copilot simplifies the creation, management, and delivery of mixed reality content using customer-defined data sources, enabling companies to adopt mixed reality much faster.

Step-by-Step Assistance

Powered by Microsoft Azure OpenAI Service, large language models allow Copilot in Dynamics 365 Guides to do far more than answer one-off questions. Workers can have progressive conversations, such as “Walk me through the steps to disassemble this filtration unit,” or “I disconnected the motor, what’s next?” Copilot follows along with the process and conversation using customer’s curated knowledge base to articulate answers with speech, text, or 3D holograms, eliminating ambiguity and reducing the need for rework.

On-demand Information

With connections to the Internet of Things (IoT) sensors, operational data, and service records, Copilot in Dynamics 365 Guides can answer questions, such as “What is the current humidity rating?” or “When was this last inspected, and were there any concerns?” HoloLens 2 allows Copilot to pinpoint specific assets, identify them, and access relevant information sources in real time. Frontline workers can access rich information to troubleshoot or repair with AI assistance in the context of their work.

Expert Knowledge Integration

Often only a more experienced worker has the knowledge required to complete a complex task or repair a vital piece of equipment. When that worker is unavailable or retires, that knowledge is lost. Copilot in Dynamics 365 Guides can leverage notes or call transcripts from previous expert activity to guide new workers through the same process. This makes it easier and faster for workers to learn from each other and automate internal knowledge transfer—bolstering workforce productivity and business agility.

Streamlined Documentation

Workers will be able to use Copilot in Dynamics 365 Field Service mobile to speed up the process of completing and documenting their work. Instead of filling out paperwork, workers can simply describe their progress to Copilot using their voice. Copilot will provide suggestions to efficiently check off service tasks, add notes, and update product quantities as well as statuses, accelerating data entry. 

Copilot in Dynamics 365 Guides transforms the real world into a prompt and an anchor for AI. Keeping human agency at the center of the product, this tool radically transforms how computing technology can help frontline workers take action in the real world. Copilot in Dynamics 365 Guides holds significant potential for businesses seeking greater efficiency and effectiveness in their operations.

Deploy Dynamics 365 Guides and Remote Assist today

The private preview for Copilot in Dynamics 365 Guides is launching today with a limited set of customers and capabilities. The upcoming public preview of Copilot in Dynamics 365 Guides will be available first on HoloLens 2, and we are planning to bring this capability to mobile devices. Interested customers can get started today by deploying Dynamics 365 Guides and Remote Assist on either HoloLens 2 or mobile devices as the first step.

Microsoft believes that when you create technologies that can change the world, you must also ensure that the technology is used responsibly. Copilot in Dynamics 365 Guides is developed in alignment with Microsoft’s responsible AI standards, and consistent with our long-standing commitment to data security and privacy, helping to ensure AI systems and solutions that are trustworthy and safe.

Learn more about Microsoft Ignite 2023 announcements for Dynamics 365 and industry transformation.

The post Introducing Copilot in Microsoft Dynamics 365 Guides, bringing generative AI in mixed reality to frontline workers appeared first on Microsoft Industry Blogs.

Source: Microsoft Industry Blog