Research on the application of deep learning to audio classification has predominantly focused on convolutional neural networks (CNNs) since 1995, when Yann LeCun achieved success in voice recognition using CNNs (LeCun and Bengio, 1995). CNN-based audio classification typically involves converting audio data into spectrograms, which represent the data in the frequency domain and along the time axis. These spectrograms are input into CNNs, which extract features with high accuracy and speed. However, CNNs have limitations in capturing the temporal characteristics inherent in audio data. In 2006, Graves et al. introduced a method for learning temporal characteristics using recurrent RNNs for voice recognition and classification (Graves et al., 2006). While RNNs are capable of processing time-series data, their ability to learn temporal characteristics diminishes when dealing with longer sequences, such as those encountered in voice recognition tasks. To address these limitations, improved RNN-based models were developed, including LSTM networks and Gated Recurrent Units (GRUs). These models demonstrated enhanced capability in processing longer time-series data. For instance, Yu et al. (2020) utilized bidirectional RNNs, which process hidden states in both forward and backward directions, in combination with attention mechanisms to classify music genres. Similarly, Gan (2021) employed bidirectional LSTMs for music feature classification. While CNNs and the RNN family of models are primarily supervised learning approaches, where correct answers are explicitly provided during training, unsupervised learning methods have also been explored. For example, autoencoders, including those connected to variational autoencoders or multi-layer perceptrons, have been applied to learn and classify data features without labeled data. These approaches have been employed to address dataset imbalance issues through data augmentation (Saldanha et al., 2022) or to generate spectrograms for robust feature extraction (Huang et al., 2022).

Among the various applications of deep learning-based audio classification, construction sites present acoustic environments similar to the crushing operations examined in this study. Scarpiniti et al. (2021) conducted research that employed CNNs and LSTM networks, an improvement over traditional RNNs, for the classification of heavy equipment sounds at construction sites. This study utilized a variety of acoustic feature extraction methods and achieved an average accuracy of 95% in distinguishing between different manufacturers of the same type of heavy equipment. The methodology involved collecting features extracted through diverse approaches and inputting them into CNNs as images and into RNNs as sequences (Scarpiniti et al., 2021).

The present research builds on audio classification techniques used in construction site contexts to classify repetitive work noise. Its objective was to enable situational assessment of remote crushing operations conducted in underwater environments. The study focused on robust classification of sounds from work sites, which vary in real time, using deep learning-based audio classification techniques. To extend the applicability of audio classification, a real-time pipeline was designed to process crushing sounds, classify operational scenarios, and validate the approach through ground-based crushing experiments.

Understanding the features of the data being processed is a critical aspect of problem-solving using deep learning. In the context of audio classification for this study, comprehending the characteristics of the collected sounds is essential, as it directly influences the extraction of appropriate acoustic feature vectors and the training of models. The acoustic features of the crushing operation, which serve as the target for classifying operational states, can be categorized into noise generated during idle operation and noise produced during active crushing. The idle-state noise primarily includes sounds originating from the hydraulic power unit (HPU) and the crushing tools, while the active crushing noise is limited to the interaction between the crushing tools and the work material, which in this study is gypsum. The HPU functions as a hydraulic supply unit, powering the hydraulic devices utilized by the robots in the CPOS. Since HPUs are commonly employed in heavy construction equipment, and the crushing sound closely resembles noises typically observed at construction sites, the methodology for this study was primarily informed by techniques used in construction site audio classification. Examination of the noise in the robot’s idle state (Fig. 3(a)) through the Mel spectrogram revealed continuous signals in the 500, 700, and 1,100 Hz frequency bands. In contrast, the noise generated during the crushing operation (Fig. 3(b)) exhibited a distinct signal at 900 Hz, recurring at intervals of 53 to 57 ms. When strong vibrations occurred during crushing, signals were observed across a broad frequency range, particularly above 2,000 Hz, as illustrated in Fig. 3(c).

Acoustic signals, as one-dimensional time-series data, require the extraction of feature vectors to enable input into deep learning model classifiers. This process involves transforming the signals to reflect the acoustic features described above. The feature vector extraction methodology, illustrated in Fig. 4, is primarily divided into two stages. In the first stage, the amplitudes of the input acoustic signals are normalized and standardized. This step addresses variations in the distance and direction of the sound source relative to the sound collection device, ensuring consistency in the input data. In the second stage, the audio samples are converted into feature vectors using an acoustic feature vector extraction algorithm. Given that the acoustic features of the crushing operation are distinguishable by the human ear, MFCCs were employed for feature extraction (Majeed et al., 2015). MFCCs leverage the Mel scale (Eq. (1)), which enhances features in the audible frequency range by representing low frequencies on a linear scale and high frequencies on a logarithmic scale. In Eq. (1), m denotes the mel frequency and f is represents frequency. For this study, the parameters were configured to optimize frequency resolution and capture signals generated during the crushing operation. Specifically, the number of cepstral coefficients was set to 40, the window length for the fast Fourier transform (FFT) was defined as 2048, and the hop length was set to 512. Based on this, the frequency resolution was increased to include the signals generated during crushing.

Given the distinct frequency-band features of the crushing sound described above, audio classification can theoretically be performed using traditional machine learning models such as support vector machines. However, the acoustic features mentioned were derived from sounds collected in terrestrial environments and cannot adequately account for variations introduced by underwater environmental factors, such as water temperature and pressure (Wenz, 1972). To address this limitation and construct models and pipelines capable of robustly classifying sounds under underwater conditions, deep learning models were employed in this study. Among machine learning approaches designed to process data-driven information, RNNs and transformer models are commonly used for time-series data. In this study, an RNN model was selected due to its lower computational cost, which aligns with the requirements of real-time operations in underwater crushing machinery. The RNN model, illustrated in Fig. 5(a), incorporates a hidden layer between input and output, enabling it to retain and reflect prior outputs in subsequent inputs. This characteristic makes it well-suited for capturing temporal contexts in real-time audio classification tasks. However, RNNs suffer from the issue of gradient vanishing, which prevents the effective propagation of initial input information to later stages in the sequence. To mitigate this problem, LSTM networks, depicted in Fig. 5(b), were employed. LSTM networks address this limitation by incorporating a cell structure to store long-term memory and gate structures to manage short-term memory. In typical speech processing applications using RNNs, signals are segmented into 20–40 ms intervals for analysis (Paliwal et al., 2011). However, in this study, longer sound segments were used, with audio divided into 100-ms intervals to ensure that signals with an average duration of 55 ms were captured. In this study, the RNN and LSTM models were trained on crushing sound data, and their classification performances were analyzed to evaluate whether LSTM provided superior accuracy compared to RNN for this specific application. The deep learning models were designed to classify task situations using audio data as input, as shown in Fig. 5(c). To enhance the classification of similar task situations, such as standard cutting and hard cutting, two RNN layers, each comprising 128 hidden units, were stacked. Similarly, a second model with two LSTM layers was constructed. For each model, a classification layer followed the RNN or LSTM layers. This classification layer was designed to predict task situations using a fully connected layer and a Softmax output layer.

In this study, a three-stage real-time prediction pipeline was developed. The first stage involved generating a dataset comprising audio files collected during the experiment and their corresponding class labels. In the second stage, an untrained deep learning model, along with learning functions and feature extraction functions, was imported, and training was conducted using the previously generated dataset. In the final stage, the audio stream, formatted in Pulse Code Modulation (PCM), was stored as a buffer within the sound collection device. The device’s sampling rate was set to 44,100 Hz, and audio samples spanning 100 ms were input into the trained model to determine the predicted class. During this process, components such as deep learning models, learning methods, and feature extraction techniques were designed as modular elements to allow for future research expansion (Fig. 6). The pipeline’s learning functions were implemented using PyTorch, a Python-based deep learning library. Additionally, the PyAudio library was utilized for acoustic feature extraction and the real-time collection of audio data.

The crushing experiment conducted to construct the dataset utilized a gypsum plate measuring 1.06 × 0.66 m². During the operation of the crusher at 100 rpm, the vertical cutting operation was configured to a depth of 5 mm, while the horizontal cutting operation was executed such that the manipulator’s endpoint moved at a rate of 0.1 to 0.2 mm/s. The generated crushing sound was recorded in the MP3 format at a sampling rate of 44,100 Hz using a GoPro^TM 8 camera, as a dedicated sound collection device could not be installed during the experiment. The recorded audio was subsequently decoded into the PCM format for use in model training. The experiments were conducted nine times, with horizontal cutting (Fig. 7(d)) performed after vertical cutting (Fig. 7(c)). Each experiment lasted approximately 6 min, resulting in a total of 57.42 min of audio data. The collected audio data were divided into 34,452 segments, each with a duration of 100 ms, to align with the audio frame required for real-time prediction. A two-dimensional dataset was generated by associating these segments with their respective situational judgment class information (Fig. 8). The dataset was partitioned into training, validation, and testing sets in a 6:2:2 ratio for model training. The architecture of the deep learning models used for this study is depicted in Fig. 5(c). To enable a direct comparison between RNN and LSTM models, the structural parameters, including the number of hidden layers and nodes within these layers, were kept consistent. Two model variants were created by altering the node type to either RNN or LSTM. For a fair comparison of model performance, the batch size, representing the number of input data instances processed simultaneously during training, was set to 128, while the epoch, defining the number of iterations over the entire dataset, was set to 10 for both models. Given that the deep learning models must perform operations within the canister mounted on the CPOS rover, they cannot rely on graphics cards. Therefore, model training was conducted in an environment featuring an Intel 13th Gen i9-13900K^TM CPU and 64 GB of RAM.

The developed algorithm was determined to operate in real time, as it could classify task situations at a rate of 10 Hz when 100 ms of acoustic data were input. Audio data, not used during the training process, were recorded using a Sennheiser^TM Profile microphone. The results of classifying situations by both a human operator and the trained model were then compared. In this case, the microphone was configured to collect data in the PCM format at a sampling rate of 44,100 Hz, consistent with the settings used for collecting training data. These results are illustrated in the graph in Fig. 9. For the manual classification of operational situations (Fig. 9(a)), the results predicted by the trained RNN model (Fig. 9(b)) achieved an accuracy of 84%, while the results from the LSTM model (Fig. 9(c)) achieved an accuracy of 88%. The prediction process for both models required a total operation time of 5 ms, which included 4 ms for extracting acoustic features and 1 ms for predicting the class. This confirmed that predictions could be performed at a maximum frequency of 200 Hz. The learning process for both the RNN and LSTM models was represented by the average loss (Figs. 10(a) and 10(c)) and the average accuracy for the test set (Figs. 10(b) and 10(d)) plotted over each epoch, showing the performance after one out of ten epochs. The performance metrics for the models’ training are presented in Tables 1 and 2 for the RNN and LSTM models, respectively. In this study, data imbalance and the influence of the base class were prominently observed.First, the models exhibited the lowest performance for the hard cutting class, as it had the smallest amount of corresponding data. Thus, the models struggled to learn this class effectively. In contrast, for the cutting class, the precision, recall, and F1 score (the harmonic mean of precision and recall) all showed high values. This was due to the abundance of training data, with 4,333 frames available for learning. Second, for the base class, the precision and recall values were relatively lower, with scores of 78 and 66, respectively, for the LSTM model. This is likely because the base class did not belong to any of the three primary classes, leading to misclassifications where other classes were incorrectly predicted. The confusion matrix derived from the test set (Fig. 11) confirmed these predictions for the base class. As previously described in section 3.2, the LSTM model is known to demonstrate superior performance compared to the conventional RNN. To verify whether this holds true for the actual learning results, we compared the performance of both models using the same hyperparameters, model structure, and data as outlined in section 4.1. The comparison was conducted through the confusion matrix (Fig. 11), and the precision, recall, and F1 score classification performance evaluation tables were analyzed for both the RNN (Fig. 11(a), Table 1) and LSTM (Fig. 11(b), Table 2). The RNN displayed distinct characteristics when compared to the LSTM model, and similar performance was observed for the idling and cutting classes, which had the largest amount of training data. However, for the hard cutting class, which is similar to the cutting class, the RNN demonstrated lower performance in both precision and recall than the LSTM, thereby confirming that the RNN is not suitable for the classification task in this study.