2 .Materials and Methods

The workflow of the developed AR-based system is depicted in Fig. 1. It consists of various steps including the preliminary virtual content preparation, the implementation of the subject tracking to register both the fixed glenoid fossa and the movable mandible, the AR overlay of TMJ in sagittal and frontal views, and the calculation of TMJ trajectories as estimated from displacement of virtual markers in AR view recordings.

Fig. 1
Fig. 1

Workflow for the AR system developed to aid surgeons in the clinical assessment of TMJ alterations

Full size image

2.1 Preparation of the Virtual Content

As first step, the anatomical virtual models were prepared starting from the subject imaging. The study involved a volunteer, having a computed tomography (CT) scan of the midface available. The skull, the mandible, and the head skin of the volunteer were segmented from CT scan using Mimics Medical 25.0 (Materialise, Leuven, Belgium), and the corresponding three-dimensional (3D) meshes were generated and saved in Standard Tessellation Language (STL) format (Fig. 2a).

The skull model was cut using 3-Matic Medical 16.0 design software (Materialise, Leuven, Belgium), so that only the glenoid fossa (blue-colored), i.e. the portion of the temporal bone that articulates with the mandible, was left (Fig. 2b). The STL file of the mandible was duplicated and the copy was symmetrically cut in a way to separate the left and the right ramus. This latter was used as virtual content to be displayed in the sagittal AR view (right side) so that to avoid the perspective overlap of the contralateral ramus if using the whole mandible.

Some virtual spherical markers (pink-colored) were added both on the fixed glenoid fossa and the moving mandible to have some reference points to follow in the reconstruction of the TMJ trajectories from video recordings captured during the AR view (Fig. 2b).

All image processing operations were performed on platform Intel® Xeon W-2123 CPU@ 3.60 GHz, 64 bit, 32 GB RAM; graphics card: NVIDIA GeForce RTX 2070 SUPER) and operating system (Windows 10, Microsoft Co., Redmond, WA, USA).

Fig. 2
Fig. 2

The anatomical 3D models reconstructed from the volunteer’s CT scan (a) and subsequent preparation of the virtual content for both the sagittal and frontal AR views (b): the glenoid fossa in the temporal bone (blue-colored), the mandible (green-colored) with the spherical markers for trajectory reconstruction (pink-colored)

Full size image

2.2 Tracking Modalities

For the presented AR application, it is required to simultaneously recognize and track two different anatomical parts involved in the TMJ, i.e., the fixed glenoid fossa and the moving mandible. So, a multi-modality tracking approach was implemented, by combining two different tracking modalities, in each case using the integrated cameras of the AR display. The AR scene was built using Unity3D v2019.4.21f1 (Unity Technologies, San Francisco, CA, USA). All the reconstructed 3D virtual models were imported into Unity3D software extended with the Vuforia Engine Software Development Kit (PTC Inc., Boston, MA, USA), version 9.8.5. Vuforia Engine uses feature tracking and matching concept of computer vision to track and detect objects and images in the real world and based on their orientation it renders the corresponding virtual objects on those.

For the glenoid fossa, which represents the fixed component of the TMJ, a marker-less Vuforia Engine tracking modality (“Model Target”), based on object recognition by its shape, was used.

For tracking the moving mandible, a marker-based tracking was implemented, by leveraging the Vuforia Engine “Image Target” and “Multi Target” tracking modes.

Working with this multi-modality tracking approach, the movement of the mandible can be tracked continuously, and its position can be displayed to the surgeon wearing the HoloLens 2 device in relation to the fixed glenoid fossa.

Two different AR applications were implemented, one for the frontal view and one for the sagittal view, in order to replicate the main views adopted during the clinical observation and kinesiography examination. During the AR view, interactable toggles (check boxes) were added to allow the user to turn on/off the rendering of each virtual structure.

Details on the implemented tracking modalities using Vuforia Engine SDK were reported in the following paragraphs.

2.2.1 Model Target

It allows to recognize objects in the real world based on their shape using pre-existing 3D models, such as a 3D CAD (computer-assisted design) model or a 3D scan of the object. The object has to be geometrically rigid and to present not shiny surface features. To implement this marker-less tracking method, it is required to generate a Model Target using the Model Target Generator (MTG) application. The MTG takes as input a 3D model representing the object to be tracked, draws an image showing an approximation of the object’s 3D shape and allows for setting up the distance and viewing angle the object should be recognized from (“Guide View”). Then, it is required that the user holds the AR display at these angle and distance relative to the real object (i.e. the set Guide View), to initialize the tracking. The Model Target tracking also works if the object is not well textured or its appearance varies – e.g. different colors or textures.

For this specific application, the 3D model of the volunteer’s head skin was used as Model Target to track the subject’s head and to provide the virtual-to-real registration of the glenoid fossa, i.e. the fixed part of the TMJ. Two Guide Views were generated for the Model Target to implement both the frontal and sagittal AR views (Fig. 3).

Fig. 3
Fig. 3

Model Target generation with both sagittal and frontal Guide Views, starting from the 3D model of the volunteer’s head skin

Full size image

2.2.2 Image Target (planar marker)

This is the most commonly used Vuforia Engine tracking method based on the recognition and tracking of a pre-defined image (Image Target) that provides sufficient details to be detected, typically a QR-code pattern. The Vuforia Engine detects and tracks the image by comparing extracted natural features from the camera image against the known Image Target previously reviewed in terms of tracking performance and stored in a cloud database. The Image Target can be downloaded from the database as a package suitable for integration in the Unity app.

2.2.3 Multi Target (cuboid marker)

This method uses multiple image targets, assigned to each face of a cube, to build a cuboid that is detectable from a device camera. Multi Target functions in such a way that all faces of the cuboid can be tracked at the same time because they have a shared pre-defined pose relative to Multi Target’s origin. This enables the entire Multi Target to be tracked when any of its face targets has been detected.

For both the Image Target and Multi Target tracking a dedicated tracker was designed and fabricated (see Par. 2.3).

For validating the novel AR application versus the standard kinesiography examination, we explored two multi-modality tracking options: (1) Model Target and Image Target (planar marker) tracking; (2) Model Target and Multi Target (cuboid marker) tracking.

2.3 Design and Fabrication of patient-specific Tracker

The 3D model of the mandible was used to design a custom dental splint tailored to fit the lower arch teeth of the subject. This splint was equipped with a holder designed for unique connection to the marker utilized to track the movable mandible (Fig. 4).

We experimented the use of both a planar marker (40 × 40 × 3 mm) with a binary QR-code pattern to implement Image Target tracking, and a cuboid marker (40 × 40 × 40 mm) which contains a unique QR-code pattern on each face to implement Multi Target tracking. To limit the weight of the cuboid tracker, it was designed as a hollow structure with a wall thickness of 1.5 mm and with a hole on the bottom face to facilitate the removal of the support material during the 3D printing process.

When using the planar marker (Image Target tracking), it was necessary to provide two versions of the tracker, one with the QR-code having an orientation parallel to the frontal plane, one with an orientation parallel to the sagittal plane (Fig. 3a, b). This ensures to have the markers constantly pointing towards the AR diplay camera when the surgeon looks at the subject from the two perspectives.

For Multi Target tracking, it was enough to design a single cuboid tracker capable of being detected from several points of view (Fig. 3c).

Fig. 4
Fig. 4

The designed tracker anchored to the subject’s teeth and provided with the planar Image Target (a, b) and the cuboid Multi Target (c)

Full size image

The splint with the holder was produced with stereolithography 3D printer (Form3B, FormLabs, Somerville, MA, USA) using a biocompatible resin (Surgical guide, FormLabs, Somerville, MA, USA), while the planar and cuboid markers were produced as textured objects using Polyjet 3D printing technology (J720 Dental 3D printer (Stratasys Ltd., Eden Prairie, MN), and then securely connected to the holder through a pin-hole mechanical joint.

2.4 Experimental Validation

For the experimental validation, the TMJ trajectories derived from the AR system were compared to the ones obtained by standard kinesiography (see 2.4.1).

The healthy volunteer, of whom the virtual anatomical reconstructions of the joint of interest were made in the preliminary preparation of the AR system (see details in 2.1), was asked to perform the TMJ task, i.e. the opening and closing of the mouth, while the surgeon wearing the AR glasses was looking at her. During the task under AR view, the horizontal and vertical excursions of the jaw movements, i.e. the TMJ trajectories, were derived by post-processing video recordings of the AR views as captured by HoloLens camera (see details in 2.4.3) and then compared to kinesiography trajectories. Both sagittal and frontal views were tested to three-dimensionally evaluate the trajectory of the TMJ during the opening and closing of the mouth. Details on this experimental phase are reported in the following paragraphs.

2.4.1 Kinesiography-Derived TM Trajectories

A commercially available kinesiography (KG) device (K7 Myotronics Kynesiograph, Harari MS, Milan, Italy) was employed to assess and record mandibular function in the volunteer. All KG assessments were conducted by a specialist with expertise in operating such devices. During the examination, the subject was seated on a chair with the trunk perpendicular to the floor and head upright. The KG recordings were performed using a magnet temporarily positioned on the subject’s buccal mucosa beneath the mandibular central incisors. This magnet monitored the mandible’s position relative to a sensor array suspended in front of the face by a lightweight frame. The frame was secured on the bridge of the nose and fastened behind the head with straps.

Under the guidance of the specialist, the volunteer executed opening and closing jaw movements, referred to as the “TMJ task.” The frame captured real-time variations in the magnetic field as the magnet moved in three-dimensional space. Utilizing these coordinates, graphic representations of TMJ trajectories were generated along both frontal and sagittal views. From these trajectories, metrics such as maximum mouth opening and maximum lateral deviations from the mid-sagittal plane during jaw opening (measured in millimeters) could be extracted.

2.4.2 AR View

The AR application was initially implemented for the Microsoft HoloLens 2 device, one of the most relevant commercial Optical See-Through (OST) HMDs. It consists of a high-definition, stereoscopic, optical HMD with a set of grayscale, RGB, and depth cameras which identify the surrounding geometries. Moreover, it includes sensor fusion algorithms which recognize hand gestures and voice commands. It represents an upgrade in terms of hardware (enhanced field of view (52°), reduced weight (566 g) and improved battery life (3 h) and software, compared with its predecessor (Palumbo, 2022).

Before starting the test, the surgeon was instructed to calibrate the HoloLens 2 HMD using the Microsoft calibration app, which relies on the user’s interpupillary distance. This calibration ensured optimal perception of the holograms. Next, the volunteer was asked to wear a dental splint equipped with the tracker and to perform the same TMJ task as during the kinesiography examination. The surgeon observed the subject preforming the task from both sagittal and frontal AR views, in order to assess the trajectory of the TMJ in three dimensions. For each view, the system simultaneously tracks both the subject’s head using the Model Target modality and the moving mandible using the Image Target modality (via QR-code markers). When passing from frontal to sagittal view it was necessary to change the planar QR-code to maintain the optimal orientation for the target image relative to the HoloLens camera.

During the procedure, the surgeon can press toggle buttons to turn on/off the rendering of the virtual elements.

The procedure was repeated with the volunteer wearing the cuboid tracker for validating the Multi Target tracking option. The AR views perceived by the surgeon, with the glenoid fossa and the corresponding moving mandibular condyles directly superimposed on the subject’s face, were reported in Fig. 5 and Fig. 6, as well as in supplementary video material (Video1, Video2, Video3, Video4).

Fig. 5
Fig. 5

The sagittal (a) and frontal (b) AR views experienced by the surgeon during the experimental validation. The fixed glenoid fossa (in blue) is registered via the Model Target tracking based on the volunteer’s head skin model. The moving mandible (in green) is detected and tracked via the Image Target (planar marker) anchored to a dental splint

Full size image
Fig. 6
Fig. 6

The sagittal (a) and frontal (b) AR views experienced by the surgeon during the experimental validation, when using the Multi Target (cuboid marker) to track the moving mandible

Full size image

The AR app was also built as an Android app for mobile devices, to be used on a handheld device such as a tablet (Samsung Galaxy Table S5E).

2.4.3 AR-Derived TMJ Trajectories

Apart from directly overlaying the TMJ of interest onto the subject’s face, we endeavoured to obtain from the AR view quantitative data regarding the TMJ trajectories to compare with the kynesiographic-derived trajectories. During the AR view, the video frames recorded using the HoloLens cameras were imported into Matlab App-Designer (R2019b, MathWorks, Natick, MA, USA), and a script was devised to identify the virtual spherical markers (the pink ones) positioned on both the moving mandible and the glenoid fossa, as well as on the forehead as references. The displacement of mandible markers between successive frames was calculated, and plots describing the mandible’s trajectory during the TMJ task, were generated for both frontal and sagittal views (Fig. 7). The reference markers placed on the glenoid fossa and forehead were used to potentially rectify any errors arising from misalignment in the virtual-to-real registration during the TMJ tasks. For both frontal and sagittal views, the TMJ task was repeated 5 times by the subject, and the AR-derived TJM trajectories at each repetition were superimposable (they can be described by the representative plot in Figs. 8 and 9, left side).

Fig. 7
Fig. 7

Example of a sequence of video frames recorded with the HoloLens camera. These frames were used to derive the TMJ trajectories from the AR view by following the displacement of the virtual spherical pink markers located on the moving virtual mandible

Full size image

3 Results

The developed AR application demonstrated be able to track and display in overlay to the subject the anatomical structures of interest for evaluating the TMJ kinematics.

The marker-less Model Target tracking worked efficiently in recognizing the subject’s head shape, and the AR markers on the dental splint were almost immediately detected in both applications (Image Target and Multi Target). The surgeon visually confirmed a correct registration based on the alignment of the virtual skin’s head model with the actual subject’s head shape, and the AR markers silhouette with the actual AR markers mounted on the tracker.

The subject considered acceptable the presence of the tracker anchored to the teeth during the execution of the task. Of the two options, the cubic tracker was perceived as more intrusive as it was heavier and bulkier.

In the following sections, we report the obtained trajectories from the AR video capture and their comparison with the kinesiography recordings.

3.1 Model Target and Image Target (Planar Marker) Tracking

For the sagittal view, the AR-derived trajectories of the opening and closing of the mouth showed a good match with the kinesiographic signals, both qualitatively and quantitively (Fig. 8). We obtained a deviation of only 0.2 mm for the vertical displacement between the closed mouth and the mouth fully open, and a deviation of 0.7 mm for the posterior extension, i.e. the horizontal displacement (Table 1).

For the frontal view, the AR-derived trajectory was qualitatively comparable to that obtained with the kinesiography, but a higher deviation was found for both the horizontal (2.1 mm) and vertical displacement (6.5 mm) (Fig. 9; Table 1). Specifically, the AR view underestimates the vertical displacement and slightly also the horizontal displacement compared to kinesiography.

Fig. 8
Fig. 8

Comparison between the mandible trajectories derived from AR view (with Model and Image Target tracking) and the ones obtained by kynesiography, in the sagittal view

Full size image
Fig. 9
Fig. 9

Comparison between the mandible trajectories derived from AR view (with Model and Image Target tracking) and the ones obtained by kynesiography, in the frontal view

Full size image
Table 1 Horizontal and vertical displacements (from closed to fully open mouth), obtained when using model and image target tracking, for both Sagittal and frontal views
Full size table

3.2 Model Target and Multi Target (Cuboid Marker) Tracking

In this case, trajectories derived from AR capture deviated more in pattern from kinesiography than those obtained with the Image Target option. However, the derived vertical and horizontal displacements were comparable with the ones obtained from kinesiography (Figs. 10 and 11). For the sagittal view we obtained a deviation of 0.8 mm for the vertical displacement between the closed mouth and the mouth fully open, and a deviation of 1.9 mm for the horizontal displacement (Table 2). In the frontal view, the obtained vertical displacement was more consistent with kinesiography (only a deviation of 0.6 mm), than in the case of planar Image Target.

Slight overestimation (3.0 mm) was found for the horizontal displacement if compared with kinesiography (Fig. 11; Table 2).

Fig. 10
Fig. 10

Comparison between the mandible trajectories derived from AR view (with Model and Multi Target tracking) and the ones obtained by kynesiography, in the sagittal view

Full size image
Fig. 11
Fig. 11

Comparison between the mandible trajectories derived from AR view (with Model and Multi Target tracking) and the ones obtained by kynesiography, in the frontal view

Full size image
Table 2 Horizontal and vertical displacements (from closed to fully open mouth), obtained when using Model and Multi Target tracking, for both Sagittal and frontal views
Full size table

3.3 Experience on a Patient

After the experimental validation of the AR app through comparison with the standard kinesyographic assessment, we experienced the new tool for the evaluation of a patient candidate for orthognathic surgery.

In this clinical experience, we used the AR app built for Samsung Galaxy Tablet S5E tablet, so that more users (i.e. the surgeon and residents involved in the diagnostic evaluation) can appreciate the AR view simultaneously. For the case, we used the Model Target and Multi Target (cuboid marker) tracking combination to avoid the possible underestimation of the vertical displacement.

The AR views with the glenoid fossa and the corresponding moving mandibular condyles directly superimposed on the patient’s face, were reported in Fig. 12 and in the supplementary video material (Video5, Video6), and the quantitative results of AR-derived TMJ trajectories were depicted in Fig. 13.

Fig. 12
Fig. 12

The sagittal (a) and frontal (b) AR views experienced with a patient candidate for orthognathic surgery, when using the Model Target and Multi Target (cuboid marker) combination for tracking the glenoid fossa (in blue) and the moving mandible (in green), respectively. A handheld tablet was used as AR display

Full size image
Fig. 13
Fig. 13

The TMJ trajectories derived from the AR views for the patient candidate for orthognatic surgery

Full size image

4 Discussion

In this study, we explore the potential of a new AR application to aid surgeons in diagnosing potential TMJ alterations that may necessitate surgical intervention. Unlike conventional clinical examination and kinesiography, this innovative tool offers the surgeon the possibility to “see-through” the patient, by viewing the underlying anatomical structures involved in the joint. Consequently, it gives diagnostic capabilities beyond those provided solely by visual inspection and palpation. Moreover, by processing the AR recordings, we can extract joint trajectories akin to those obtained through kinesiography. Notably, this tool holds promise for post-surgical assessment of TMJ in patients undergoing orthognathic procedures.

Our study findings confirm the efficacy of the implemented multi-modality tracking approach, which enables the simultaneous tracking of the two anatomical components involved in the TMJ. Specifically, marker-based tracking methods (such as Image Target and Multi Target) did not interfere with the 3D shape recognition of the subject‘s head during the marker-less Model Target tracking.

While marker-less tracking offers advantages by eliminating the need for cumbersome trackers on the patient, marker-based tracking remains essential for capturing the movements of the mandible, which cannot be tracked externally as a fixed 3D shape.

Furthermore, our analysis successfully computed joint trajectories from the AR video recordings, giving quantitative data similar to what achieved with kinesiographic examinations. Only in the frontal view, when using the planar Image Target, we observed a limited capability of the system to correctly track the entire mandible vertical movement. However, when moving to the cuboid marker, the entire excursion of the vertical trajectory is recovered.

In the case of patient experience, the AR tool helped in detecting and quantifying a sub-optimal TMJ kinematics. Indeed, the AR-derived trajectories showed a reduced excursion compared to the physiological one, in both the horizontal and vertical movements of the mandible (Fig. 13).

4.1 Limitations

In the study, we observed that the planar Image Target has the limitation of not allowing the complete tracking of the entire vertical movement of the mandible in the frontal view. Indeed, at the position of the maximum opening of the mouth, the AR marker surface with the QR-code no longer appears in a frontal view with respect to the HoloLens camera, so tracking is lost. This explains why the AR-derived frontal vertical excursion is underestimated if compared to the corresponding kinesyographic trajectory.

To overcome this limitation, we tested the cuboid Multi Target, which should ensure to have the target image visible throughout the entire excursion of the joint. And indeed, the derived maximum vertical excursion in the frontal view seems to be complete, as demonstrated by comparison with kinesiography. As a counterpart, the cubic marker is heavier and bulkier. This may have caused some unnatural movements of the joint that can explain the slight deviations in the trajectory pattern we found when using the cuboid Multi Target (Figs. 10 and 11). On the other hand, the pattern obtained with the planar marker is more natural and comparable to that obtained with kinesiography (Figs. 8 and 9). In the light of our findings, a good compromise for the application could be to use a smaller cuboid marker, which can still guarantee stable and continuous tracking, without being too intrusive.

As we previously reported for a phantom study (Cercenelli et al., 2021; Ruggiero et al., 2023), also in this case we observed that, especially for the marker-less Model Target tracking, the registration accuracy is quite sensitive to ambient light conditions. So, it was necessary to make several attempts, optimizing the light conditions, before obtaining an acceptable and stable registration of virtual-to-real head of the subject.

Another limitation is the latency in tracking the moving mandible, i.e. the delay between when a pose of the mandible changes and the AR display changes in response. Indeed, a too-fast opening/closing of the mouth causes a loss of tracking of the moving mandible, therefore errors in virtual-to-real registration and consequent unreliable AR-derived movements. For this reason, it was necessary to ask the subject to perform the TMJ task quite slowly. Moreover, in this current preliminary version, the proposed AR system cannot provide a real-time calculation of the AR-derived trajectories but requires a post-processing phase, which is however automated. From the user’s side, the HMD allows for a visualization consistent with the surgeon’s point of view, although the holograms were perceived poorly visible. When using the tablet, a particularly appreciated aspect was the brightness of the holograms.

4.2 Future Directions

Future improvements can be expected for the project. First the design of the cuboid tracker should be optimized in order to make it lighter and less intrusive to the subject.

Moreover, it could also be useful to integrate the AR application with additional visual cues that may further help the surgeon in the diagnosis process. For example, a video recording of the physiological movement of the TMJ in the frontal and sagittal view can be projected in the right-up angle on the HoloLens field of view. In this way, the surgeon during the AR-assisted clinical examination can use it as reference. Moreover, future efforts will be addressed to provide the surgeon with more real-time trajectory data for timely decision support.

A notable aspect of this methodology is its adaptability to various surgical tasks requiring simultaneous tracking of multiple objects. For example, in orthognathic surgery, a common task deals with bone segments repositioning, such as the jaw, according to a predefined preoperative plan after they have been cut. In such cases, marker-less tracking through facial 3D shape recognition can be utilized to show the target jaw position onto the patient, such as by displaying a cubic volume encompassing the jaw part in its planned position. Real-time tracking of the cut jaw can be achieved using marker-based tracking, with a tracker anchored to the teeth, as demonstrated in the presented application. The surgeon, wearing the HMD, can view the planned target volume for repositioning, superimposed on the patient, while tracking the resected bone part in real-time using marker-based tracking. The task of bone repositioning consists of aligning, under AR guidance, the virtual cubic volume associated with the movable cut jaw with the virtual cubic target volume projected onto the patient.

Similar methodologies could be considered for bone repositioning tasks in other surgical fields, such as orthopedic surgery.

5 Conclusion

In this research, we developed an augmented reality application utilizing multi-modality tracking to aid surgeons in diagnosing potential TMJ alterations. Two different options for multi-modality tracking were evaluated in an experimental setting, with the standard kinesiography examination serving as a reference for quantitative assessments. The combination of marker-less Model Target tracking with cuboid Multi Target marker-based tracking emerged as the most promising option. This approach holds promise for easy integration into clinical practice in the near future, albeit requiring further refinement in certain aspects. In comparison to kinesiography and standard clinical examinations, the proposed AR application offers the advantage of directly visualizing the patient’s anatomy and observing the joint in action superimposed on the patient’s face. This preliminary experience can serve as a source of inspiration for future advancements in AR applications requiring similar multi-modality tracking approaches to simultaneously monitor two anatomical parts, as for example in the context of maxillofacial and/or orthopedic surgery for intraoperative bone repositioning.