Magic Leap 2 Platform

Company: Magic Leap
Role: Sr Product Designer
Type: Unity, Android, Godot engines
Tools: Figma, Bezi, After Effects, Google Suite
2021-Present

Overview

During my 3+ years at Magic Leap, I have been a part of 12+ release cycles shipping critical platform features for the Magic Leap 2. Designing for a one-of-its-kind AR headset and hardware running multiple engines has brought really interesting UX challenges.

Key Features and Takeaways



Controller mapping diagram

Controller (2021-2022)
UX: System Button Mapping, Shortcuts, Hardware Interactions, Haptics

One of my first takeaways working on a hardware product was learning about the impacts of locked hardware designs on a platform UX experience. I was in charge of system button mapping and haptics for the controller, for which the system interactions were optimized for.

On the controller, there is limited button real estate, and limited positions for them. One memorable issue we had was that most users had trouble performing a combination button shortcut (Home button and Bumper button) to take a video or photo. Because the placement of the Home button was in an un-intuitive and ergonomically uncomfortable, but electronically dependent spot, and we were constantly running into debates about what the spare button above it would do and whether we could use it instead; it was more comfortable to reach. However, here I learned one important thing: the platform functions are not the most important user experience to an end user. Platforms are in service to the applications that run on it. After much debate, it was decided that the button would be left largely unused by the platform and be available mainly to developers to map however they choose. 


Screenshot of what System Capture looks like overlaid over the Home menu

System Capture (2021-2022)
UX: End-to-end design and integration and improvements over 5 releases

Designing a camera application where the viewfinder is what you see through the headset, makes you think about real hardware challenges like limits of the field of view of the headset, as well as its follow behavior and overlay position over other content and the real world. Although there were plenty of shortcuts (voice commands, controller keys) our main challenge was designing an interface for a controller ray, our customers’ main input. We went through so many iterations trying to figure out where to position buttons and settings menus in a way that wasn’t annoying during a capture or resulted in false positives. Another challenge was balancing the needs of power users and novice users to AR. A frequent decision I always had to defend was why I had a 3 second countdown when one selected the camera button; folks used to the camera had become power users and found it annoying. However, as lead on this project I had to consider people new to the concept of capturing video with your head--when I tested the app on non-technical folks, they needed time to adjust to their content and surroundings, and the timer was helpful. With system applications, it’s important to have a default that will work for everyone, and give the option to power users to make modifications. 


Custom Fit introduction

Custom Fit (2022-2023)
UX: End-to-end design and integration, redesign efforts and testing

The Custom Fit Unity application helps users wear the device correctly using proper nose and forehead pads. Once you’re in a good fit, you can also calibrate the headset to your eyes and improve other features like hand tracking and IrisID.

The calibration applications I’ve worked on have been the most challenging, because at their essence, they are trying to smoothen hardware friction with software. For example, to wear our headset properly, it needs to be worn at a tilt and with nose and forehead pieces adjusted to your face shape. This is not intuitive–most people want to wear the headset like glasses and call it a day. So designing an app that instructed the user to perform physical actions - like adjusting their and taking out fit pieces as part of an application flow - was full of friction. It didn’t matter how good the instructions were, or how accurate the animations were, or how we solved the take-off-put-back-on flow. Peoples’ perception of what is a good fit is on something they’ve never worn before is personal and subjective, and most people will just ignore it.

When we saw that users were mindlessly skipping through the application and their fit wasn’t improving, the product team, eye tracking team, user research and product design took a step back and considered other approaches and had a cycle of fast prototypes and quick dogfooding studies. After our roadmap pivoted and we had taken some time to learn more about our use cases, we eventually decided to change our success metrics, and made the entire process more interactive and shorter, completely changing how we did the fit process for most users. All in all, for this project, I realized there’s a limit to what UX can fix, no matter how engaging it is. Sometimes we have to totally start from scratch and make sure you’re trying to fix the right problem and giving the right success metrics!

While I can’t publicly share these new approaches, I can share that one of the prototypes involved me explaining how to wear the device in a video. This kicked off a completely new effort to create a series of product marketing videos that explain how to wear the Magic Leap 2: 



(It’s a no-brainer at most companies who have the budget already in place, but for a resource-constrained product, it was a big win when I successfully got buy in through some quick prototyping and light testing.) 


Diagram of segmented dimming panel

Segmented Dimming (2021-2022)
UX: End-to-end feature, research, system settings

Segmented Dimming may be the most differentiating patented display technology in the AR headset market. It consists of two display panels that allowed users to dim the whole display (like sunglasses), dim only some content (we called it segmented dimming) and it also functioned with respect to an auto-brightness display feature. This was totally novel technology in the market and calculations that we were still trying to understand. Specifically, how it improved content display vividness, legibility, and how environments and 2D-3D content worked together with it.

After some studies and collaboration with our SDK team, I helped release the Segmented Dimmer Design Guidelines to help developers incorporate these different types of dimming into their apps. Designating blended dimming mode system presets based on user research is as far as we got on the platform side; its real use cases would be expanded on by developers in their applications. When there’s a new/novel application that comes out in the world, designers don’t need to know all the answers. It’s best to let creators play with it and discover its best use cases and give feedback on what to improve



Bootup LED pattern on the Compute Pack

LEDs (2022)
UX: Redesign of Compute Pack, Controller, and Headset LED patterns 

After our first release, customers complained that our LED system was confusing and non-intuitive. I worked on a redesign with our hardware and inputs team, plus a motion designer to simplify the patterns and provide easy to remember functions. It was a long tetris game of negotiations on brand palette, thermal and power requirements, 60601 compliance, and hardware limitations too technically complicated to explain here. Tip: Definitely work out your UX before you lock the hardware design!


Main menu of the Fundamentals app

Learning System + Fundamentals (2023)
UX: Design of learning system strategy and related applications

As the platform matured, we began to think about large scale deployments and multiple end-users using this headset in enterprise training scenarios. My focus in the last year has since been on first time experiences and onboarding/setup, making the general usability of the ML2 platform inclusive and accessible for as many people as possible.  I’ve been spearheading a learning design system that consists of patterns like tooltips, contextual 3D nudges, tutorials, and an immersive learning app called Fundamentals to deliver the best AR onboarding experience possible. With hand tracking becoming a new default input on our platform and system applications maturing, all of these learning efforts have been critical to smoothening the transitions for our customers who are still new to this paradigm, but they’ve also been a hit in our demo scenarios! Investment in learning experiences always increases adoption of your product in unintended points`.




Screenshot of 2024 VPAT

Accessibility (2021-current)
UX: Co-leadng the strategy and representing the platform design team with third parties

Lastly, since joining Magic Leap, I’ve been leading accessibility efforts from the design side. For anyone unfamiliar with trying to kickstart accessibility at small companies, accessibility is usually not prioritized or given much internal resources. In a tough AR market, Magic Leap has been maturing slowly, and accessibility is a mission value, though not its first priority. However, as Magic Leap 2 expands to government sectors and more, a VPAT becomes more important. Recently, I worked with product, compliance, and a third party vendor to perform an accessibility audit of the platform and generate a VPAT. From this engagement, I’ve been strategically co-presenting and educating different groups across the company on accessibility best practices the feedback for where we need to improve our efforts and working with our product team on how to integrate features in the roadmap. There’s a world of unchartered territory for augmented reality headsets and inclusive design practices!



Conclusion


Each of these features and efforts have grown my skills to:

  • Consider the end-to-end journeys for multiple customer personas
  • Design holistic interactions between multiple system settings, apps, and engines
  • consider new inputs like headpose, eye and hand tracking
  • Clarify what it means to design for AR enterprise use cases
  • Use effective communication and project management skills to work with product managers, engineers, QA, and hardware teams
  • Refine my storytelling to influence and get buy-in on new approaches
  • Fit an entire design, prototype, and user study process in agile timelines
  • Motivate teams to act quickly and in favor of user experience


Startups, limited resources, and emergent technologies have their challenges in a constantly changing industry, and I’ve learned A TON. I’ve been part of a small team learning new tools and possibilities for this medium, and I’m grateful to be contributing to the knowledge base alongside a team of other designers, user researchers, prototypers, and audio team in this journey.