Microsoft 365 Copilot, a generative AI integrated into Microsoft applications, promises a wide range of functions across the MS Office application suite. Depending on the Office application, Copilot can create documents and presentations, summarize meeting notes, perform analytics, and much more with a single click – at least if you believe Microsoft’s claims.
During the first Copilot presentation, Microsoft showed some impressive use cases. But how well do they work in the real world? We at empower® put Microsoft 365 Copilot to the test and we’d like to share our findings with you. In this article, we will focus on Copilot in Outlook and Teams.
The test framework
Ten empower® employees spent a month testing Microsoft 365 Copilot. They tested numerous features, both according to the use cases communicated by Microsoft and on the basis of company-specific requirements.
In this report, we present the test results for three applications: Teams and Outlook. Each task was accomplished by typing an appropriate command into a chat window, also known as a prompt. You’ll see how we formulated the prompts in detail and how well or poorly Copilot responded to them.
First, an important note: for this test, we used the first released version of Copilot from November 2023. This is important because we can expect regular updates in the future, which should improve Copilot continuously.
Copilot in Outlook
Outlook also focuses on creating and editing texts. Copilot can make your work in Outlook a lot easier, especially considering the large number of emails you receive and send every day.
1. Summary of email threads
Copilot provides the “Summary by Copilot” feature built into Outlook to summarize long email threads.
When we tested it, the summary created by Copilot gave a good overview of the email thread and also included brief information about the people involved. The summary was in German, since the emails were written in German. However, it’s questionable whether any summary can replace reading the individual emails in the thread, as there’s no guarantee of completeness and accuracy. Nevertheless, this feature is potentially useful, especially for people who work in large companies and are often involved in long email threads.
2. Optimization of an email body
In the second use case, there was no need to enter a prompt because Copilot already provided the ability to compose an email. We wanted to make our email more promotional.
The result was impressive. Copilot wrote a well-structured email that met our expectations.
3. Writing an email with information from a website
In this test, we asked Copilot to generate a new email based on the content of a website, namely the Wikipedia entry on Microsoft 365 Copilot. It gave us a draft email that provided a comprehensive overview of Copilot and a solid basis for further work.
“Draft a new email describing the content from https://en.wikipedia.org/wiki/Microsoft_Copilot”
4. Replying to emails
Instead of writing a new email ourselves, we instructed Copilot to reply to a specific email in the next step.
“Draft an answer in English telling my colleague about advantages of their new product called Microsoft Copilot”
The generated email was an excellent starting point, but it wasn’t ready to send. In this instance, Copilot focused on coding and software development, which didn’t totally answer the brief. However, we were generally satisfied with the result and only slightly adjusted the email and added an additional paragraph. That experience highlights how important it is to carefully examine anything Copilot provides.
Overall, Copilot in Outlook is a competent assistant that helps us summarize and respond to emails.
Copilot in Teams
Anyone who regularly works in Microsoft Teams knows how hectic the environment can be and how easily information can get lost or overlooked. Can Copilot help us structure content in Teams, make it more accessible, and organize to-dos?
1. Summary of a chat
The ability to summarize Teams conversations is one of our favorite features. You can choose what content you want to summarize and over what time period. We wanted to create a summary of a particular chat. You can see the result in the chat area on the right.
“Summarize this chat”
The summary gives a good overview that makes it easy to quickly assess the relevance of the chat content, especially for people who weren’t directly involved in the conversation. But be careful: relying solely on the summaries can lead to a loss of information. For a full understanding, it’s still essential to read the full chat history.
2. Create a presentation of a Teams summary
Wouldn’t it be handy to automatically create a presentation from this Teams summary? We tested this as well.
“Create a PowerPoint presentation that contains the information from your summary”
Unfortunately, Copilot couldn’t comply with our request. However, the answer it gave clearly explains why and clarifies what Copilot currently does and doesn’t do.
3. Reply to a chat message
What about the ability to reply to a chat message?
“This chat is about the climate change. Write an answer in which you identify the main challenges in dealing with climate change”
This was just as easy in Teams as it was in Outlook. Copilot’s reply to the previous message also appeared on the right side of the chat bar. This is helpful, as there’s no risk of accidentally sending the unreviewed message directly. Interestingly, there was no copy and paste button, so the message had to be copied manually. However, the content of the message was detailed and satisfactory.
4. Extract tasks based on a Teams conversation
Extracting tasks from a Teams conversation was also featured in Microsoft’s marketing video. We tested this feature as well.
“Which tasks do I have?”
When based on a simple chat history, Copilot answered the question without any problems. We also ran the test with more complex chat histories and meetings and were satisfied with the results. But there’s no guarantee that Copilot will actually list all the tasks that need to be done when it reviews a chat history.
We did this test not only in English, but also in German. Unfortunately, the summary didn’t work in German, neither in this use case nor in the simpler chat process before.
Similar to Outlook, Copilot works well in Teams to help you reply to messages and keep track of chats and the tasks that result from them.
Our conclusion
Microsoft 365 Copilot is especially helpful when creating or editing text. Whether it’s researching, summarizing, or rewriting, Copilot handles these tasks very well.
However, we have yet to see it revolutionize everyday office life. Many of the use cases presented by Microsoft and their results didn’t pass our real world tests. When it comes to more complex queries, the AI still too often reaches its limits. The same goes for adherence to brand specifications.
The ability to summarize in Teams and Outlook should be a great help to many people. Copilot is therefore a valuable assistant for people who work a lot with Outlook and Teams.
In Word, similar to Outlook and Teams, Copilot can assist by paraphrasing, summarizing, or giving text a different style.
Working with Copilot has been a pleasure and has given us an interesting taste of what is to come. The potential is considerable and we’re excited to see what improvements will come with the next updates from Microsoft.
Would you like to find out more about Microsoft 365 Copilot in other applications? We have also tested Copilot in PowerPoint, Excel and Word.
Microsoft 365 Copilot report: PowerPoint
Microsoft 365 Copilot review: Excel & Word
