Fiber-reinforced plastic (FRP) molded products, known for their lightweight and high-strength characteristics, are increasingly being adopted, particularly in transportation equipment such as automobiles and aircraft. The growing reliance on bio-derived fuels and electrification to achieve a carbon-neutral society has further driven their use¹⁾. Practical molded products are required to simultaneously satisfy multiple physical properties, including mechanical properties, thermal properties, environmental durability, and processability. Meeting these demands necessitates the optimization of highly complex conditions. In FRP molded products, multiple target properties are realized through intricate interactions among the resin properties, fiber properties, resin-fiber interfacial interactions, thermal history during the molding process, and the flow behavior of the fiber-reinforced resin. Consequently, optimizing the kneading and molding conditions requires substantial effort, leading to an industry-wide demand for a reduction in development time.

　Material development and process optimization have advanced in recent years through data-driven approaches such as materials informatics (MI).²⁾ Among these, the fundamental concept of predicting the diverse physical properties of complex materials using multimodal AI in the field of materials chemistry, as proposed by Muroga, Hata, and colleagues at the National Institute of Advanced Industrial Science and Technology (AIST), has been gaining increasing recognition.^3-5) The realization and dissemination of a wide variety of AI are being promoted, with the technology increasingly being considered in various industrial fields other than the materials field.⁶⁾ One such initiative is led by Konica Minolta, which is advancing the industrial and commercial implementation of digital transformation (DX) solutions by integrating its proprietary industrial measurement technologies. In collaboration with AIST, Konica Minolta is engaged in the research and development of multimodal AI, using FRP as a model material.

　This report introduces multimodal AI technology that integrates measurement data capturing the internal structure of FRP from multiple perspectives and predicts various mechanical and thermal properties. Additionally, application examples of this technology are presented.

　Fig. 1 illustrates the envisioned advancements in FRP research and development through the application of multimodal AI. The multimodal AI system was constructed by measuring various molded products from clients using multifaceted analytical methods and inputting the resulting data into the AI model. This approach enabled the AI to propose innovative solutions that would be challenging for human engineers to derive, such as optimizing processing conditions and identifying the root causes of product defects. This technology represents a framework that maximizes the utility of both measurement data and AI to generate actionable knowledge for material and molding engineers. Furthermore, it is expected to facilitate the establishment of a novel supply chain that delivers benefits to all stakeholders involved in FRP manufacturing and development.

　We have produced and evaluated a polycarbonate resin-based FRP composite material to demonstrate the concept of multimodal AI in FRP. We produced dumbbell-shaped specimens (JIS K7139 A1 type) and V-notch Charpy specimens (JIS K 7111) by compounding several different additives with a short fiber masterbatch and injection molding. The fibers used here were carbon fiber and glass fiber, and the additives were antioxidants, UV absorbers, and plasticizers. We conducted multifaceted structural evaluation and physical property measurements of 80 conditions of molded products with conditions added starting from an orthogonal array, and we used these to construct the multimodal AI (Fig. 2).

　We used six different multimodal structural evaluation techniques as inputs to the multimodal AI, consisting of the following: X-ray diffraction, infrared spectroscopy, excitation-emission matrix (EEM), X-ray Talbot-Lau interferometer,^7,8) ultrasonic measurement, and optical microscope observation. These measurement methods can capture different structural information, and each has sensitivity to complementary information such as crystallinity, functional groups, electronic state, fiber orientation, density, and surface morphology. We also selected five types of mechanical and thermal properties of the molded product, namely, specific strength, specific elastic modulus, elongation at break , impact strength, and heat deflection temperature.

　The constructed multimodal AI considers the material conditions as input, quickly generates multiple structural information using multiple generative models, and integrates these to predict physical properties (Fig. 3). Here, we sought to generate structural information from material conditions by using a conditional variational autoencoder (C-VAE)⁹⁾ for spectral data; and a conditional generative adversarial network (C-GAN),¹⁰⁾ specifically the BigGAN¹¹⁾ model, for image data. Our predictions of the physical properties of the final molded product involved a unique design of a model structure that outputs predicted physical properties through a layer that extracts features for each different structural information, enabling the implementation of the entire process from inputting material conditions to generating structural information and predicting physical properties, all in one process.

　We introduce a generative model that establishes a connection between material conditions and structural information, representing the first stage of the multimodal AI model we have developed. In this study, the initial step was to verify whether the generative model’s output accurately reproduced the measured data. Fig. 4 presents the results obtained by randomly selecting multiple conditions from the training dataset and generating X-ray diffraction spectra and optical microscope images of the surface using the trained C-VAE and C-GAN models. The generated data were then compared with the corresponding measured data. The similarity between the generated X-ray diffraction spectra and optical microscope images and their experimentally obtained counterparts confirmed the validity of the data generation process.

　Subsequently, X-ray diffraction spectra and optical microscope images of the surface were generated using the trained C-VAE and C-GAN models under representative conditions with varying fiber concentrations. Fig. 5 illustrates the changes in the generated X-ray diffraction spectra corresponding to different concentrations of carbon fiber and glass fiber. The model successfully captured the shift in the diffraction peak of the carbon (002) plane near 25° for carbon fiber, as well as the overall intensity variations associated with the amorphous structure of glass. Similarly, the generated images with different carbon fiber concentrations (Fig. 6) reflected variations in surface morphology and shading, accurately representing spatial density and fiber arrangement characteristics. We propose that enabling a generative model to learn such structural characteristics represents a novel approach to digital transformation, allowing for the differentiation of material conditions and the prediction of physical properties.

　The latter half of the multimodal AI system developed in this study rapidly generates predicted values for mechanical and thermal properties. To assess the accuracy of these predictions, leave-one-out cross-validation (LOOCV) was used. In this approach, a single sample is excluded from model training under a given condition, and predictions are made for both the trained and excluded samples. By systematically varying the combinations of training and excluded samples across all conditions, the accuracy of the predictions was evaluated. The results demonstrated that the coefficient of determination (R-squared) for specific strength, specific modulus, and impact strength was ≥0.9, whereas those for elongation at break and heat deflection temperature were slightly lower (Fig. 7). Although some variability was observed in the measured material properties, the model exhibited sufficient accuracy for selecting candidate conditions. Therefore, computations were performed using the trained multimodal AI to optimize conditions, as described below.

　Several scenarios were considered for using the multimodal AI, and the resulting candidate conditions, derived from an extensive set of computer-generated predictions, are presented below. A total of 10,000 conditions were predicted using the trained multimodal AI, and Fig. 8 illustrates the relationship between specific strength and elongation at break for a subset of these predictions. Conducting physical experiments for all 10,000 conditions is impractical; however, the multimodal AI facilitated an extremely rapid and comprehensive search. As demonstrated in the computational environment used in this study (CPU: Intel Xeon GOLD 6128, 12 cores; CPU memory: 96 GB; GPU: NVIDIA RTX A6000, 48 GB), these computations were completed within two minutes. The trends in the predicted values, as visualized in Fig. 8, indicate a reciprocal relationship between specific strength and elongation at break, with a distinct upper boundary forming a Pareto solution. In FRPs, brittle fracture is the dominant failure mode, and the required level of material toughness must be adjusted based on the intended application. The multimodal AI computation provides results that enable the estimation of attainable strength according to a specified target toughness, making it a valuable tool for condition optimization. Fig. 9 presents the results of a verification experiment, in which 12 levels were selected from the Pareto solution. The predicted values closely aligned with the experimentally obtained values, confirming that different materials could be successfully developed based on AI-generated predictions.

　Finally, we conducted analysis to interpret the contribution of structural information to the prediction of physical properties. Although methods for quantifying the relative contribution of structural information also exist,⁴⁾ we conducted SHAP (SHapley Additive exPlanations) analysis,¹²⁾ which is a statistical method based on game theory, to quantify the local contribution of each position of structural information. The use of the SHAP value enables the quantification of the position of each spectrum as well as locations in the image that have a large contribution to the target physical property. An example of this analysis is presented in Fig. 10, which illustrates the computational results for specific strength, specific elastic modulus, and impact strength based on X-ray diffraction spectra and optical microscope images. The results provide valuable insights for materials researchers, revealing features such as peaks attributed to carbon and high contribution values at the fiber–resin interface in the images. While multimodal AI-based analysis of material condition dependencies does not always yield fully interpretable results, the integration of predictive outcomes with experimental measurements facilitates meaningful discussions from a materials chemistry perspective. This approach allows researchers to derive actionable insights for condition optimization while fostering a deeper understanding of the underlying material properties.

　This report presented a research and development initiative in which Konica Minolta and AIST jointly developed a multimodal AI system—an advanced technology that enables DX in predicting various physical properties of FRPs. The developed technology serves as a cornerstone for the industrial application of analytical solutions, using FRPs as a model system. Moreover, it is expected to establish a new framework that seamlessly connects key stakeholders across materials, processes, measurement, and AI. Looking ahead, efforts will focus on enhancing the sophistication and accuracy of the model, as well as exploring novel applications. These advancements aim not only to drive DX in materials chemistry but also to contribute to the realization of a decarbonized society and a circular economy.

　The development of the multimodal AI presented in this study was made possible through the invaluable guidance of researchers, including Director Kenji Hata of the AIST Nano Carbon Device Research Center, as well as the support of relevant departments at Konica Minolta. We sincerely express our gratitude for their contributions.

　This report is a preliminary presentation of the following presentation made at the 35th Annual Meeting of the Japan Society of Polymer Processing on June 20, 2024, with revisions and additions made to the text and figures, and with permission from the Society for reprinting. We would like to thank the Society for approving the reprint request.

[C-201] Multimodal deep learning for digital transformation of fiber-reinforced plastics

*Ko Osawa1, Michihiro Okuyama1, Yusuke Araki1, Yukako Taka1, Shigeru Kojima1, Akihiro Naruke1, Shun Muroga2

1. Konica Minolta, Inc., 2. National Institute of Advanced Industrial Science and Technology (AIST)