Abstract
Sex differences in pain perception have been extensively studied, but precision medicine applications such as sex-specific pain pharmacology have barely progressed beyond proof-of-concept. A data set of pain thresholds to mechanical (blunt and punctate pressure) and thermal (heat and cold) stimuli applied to non-sensitized and sensitized (capsaicin, menthol) forearm skin of 69 male and 56 female healthy volunteers was analyzed for data structures contingent with the prior sex structure using unsupervised and supervised approaches. A working hypothesis that the relevance of sex differences could be approached via reversibility of the association, i.e., sex should be identifiable from pain thresholds, was verified with trained machine learning algorithms that could infer a person’s sex in a 20% validation sample not seen to the algorithms during training, with balanced accuracy of up to 79%. This was only possible with thresholds for mechanical stimuli, but not for thermal stimuli or sensitization responses, which were not sufficient to train an algorithm that could assign sex better than by guessing or when trained with nonsense (permuted) information. This enabled the translation to the molecular level of nociceptive targets that convert mechanical but not thermal information into signals interpreted as pain, which could eventually be used for pharmacological precision medicine approaches to pain. By exploiting a key feature of machine learning, which allows for the recognition of data structures and the reduction of information to the minimum relevant, experimental human pain data could be characterized in a way that incorporates “non” logic that could be translated directly to the molecular pharmacological level, pointing toward sex-specific precision medicine for pain.
Introduction
Sex specific differences in pain perception have been extensively studied and reviewed1,2,3. A literature search of the PubMed database at https://pubmed.ncbi.nlm.nih.gov/ on November 6, 2022 yielded 22,322 hits, with increasing total number and relative number to publications on pain (Fig. 1). The earliest hit in this search was a paper on sex differences in the apperceptive background of pain experience published in 19594, in which it was already mentioned that “females were generally predisposed to see more pain—both Intensity and duration”. Thus, the sex difference in pain seems to be well studied, and the higher sensitivity of females seems to be the most common finding.
However, statistical significance does not equate with relevance. In the present analyses a first working hypothesis for addressing the relevance of observed sex differences was the reversibility of the association. If pain thresholds differ to a relevant degree between sexes, then the mutual association of pain with sex should be reversible. It should be possible to learn sex from the pain thresholds of a given sample of men and women and then generalize this skill to infer sex from the pain thresholds of new subjects. This formulation of the question of sex differences in pain is consistent with the definition of machine learning8. Having transferred the sex problem from statistics to machine learning, a second hypothesis arises that considers the need, typical of machine learning, to filter out the “needles” from the “haystack”, i.e., to (only) use the information relevant to the problem from complex data sets. The ability to learn sex from pain thresholds may not require or be possible with every type of pain threshold, but may be (only) possible for certain types of pain thresholds. Thus, a second working hypothesis was that sex differences in pain thresholds could be used for future precision medicine approaches based on the difference in information content of thresholds for specific stimuli to infer sex. Ideally a logical “NOT” connection should be identified in the sense that sex would be related to pain threshold for one type of stimulus “but notâ€� another. This might be taken directly to a molecular hypothesis that uses the same “NOT” link to identify a molecular sex-dependent mechanism.
These hypotheses were addressed in an available dataset9 containing thresholds for various noxious stimuli applied in an experimental setting to healthy young subjects. A machine learning workflow was used as described in10 to identify a structure in the data that reflects the sex segregation of pain thresholds and that can be used to reverse the association between sex and pain to infer a person’s sex from the individual pattern of thresholds to different pain stimuli. This used algorithm building as usual in machine learning; however, the goal was not to develop an automated tool for detecting sex based on pain thresholds, but rather to use machine-learning for knowledge discovery as outlined recently10. Thus, the following report combines several types of machine learning methods to address the problem of sex differences in pain in a data set obtained from healthy individuals of both sexes in a human experimental setting. One goal was to identify distinctive types of pain that would provide a basis for translating the findings into a molecular context of nociceptive targets, potentially allowing sex-specific approaches to analgesic drug therapy.
Methods
Pain related data
The acquisition of pain-related data has been described in detail previously9. Briefly, the study was conducted in accordance with the Declaration of Helsinki on Biomedical Research Involving Human Subjects and was approved by the Ethics Committee of the Department of Medicine, Goethe University, Frankfurt am Main, Germany (approval number 150/11). Informed written consent was obtained from each of the participants, who were 125 unrelated healthy Caucasian volunteers (69 men, 56 women, aged 18–46 years, mean 25 ± 4.4 years) in whom pain thresholds to thermal, mechanical, and electrical stimuli were determined in nonsensitized and sensitized skin areas of the forearm in an open, nonrandomized design.
The study assessed pain thresholds defined as “the least experience of pain which a subject can recognizeâ€� (http://www.iasp-pain.org) and implemented in the study as the physical strength at which the answer to the question “does it hurt?” changes from “no” to “yes”, which is why the units of pain thresholds are the units of physical strength of the applied stimuli. Thermal stimuli comprised heat and cold. Heat stimuli were applied using a 3 × 3 cm thermode (Thermal Sensory Analyzer, Medoc Advanced Medical Systems Ltd., Ramat Yishai, Israel) placed onto the skin of the left volar forearm. Its temperature was increased from 32 °C by 0.3 °C/s until the subject pressed a button at the first sensation of pain, which triggered cooling of the thermode by approximately 1.2 °C/s. Heat stimuli were applied eight times at random intervals of 25–35 s. The median of the last five responses was defined as the heat pain threshold because in previous experiments a plateau was reached after the first three measurements. Cold stimuli were administered in a similar manner, except that the temperature of the thermode decreased by 1 °C/s from 32 to 0 °C. Five repetitions were used. The cut-off temperatures of the stimulation device were 0 °C and 50 °C. Mechanical stimuli comprised blunt and punctate pressure. The former was exerted perpendicularly onto the dorsal side of mid-phalanx of the right middle finger using a pressure algometer with a circular and flat probe of 1 cm diameter (JTECH Medical, Midvale, USA). The pressure was increased at a rate of approximately 9 N/cm2 per second until the subject reported pain. The procedure was repeated five times every 30 s. Punctate pressure was exerted onto the left volar forearm using von Frey hairs (0.008, 0.02, 0.04, 0.07, 0.16, 0.4, 0.6, 1, 1.4, 2, 4, 6, 8, 10, 15, 26, 60, 100, 180, 300 g; North Coast Medical Inc., Morgan Hill, CA, USA). Electrical stimuli were applied using a constant current device (Neurometer® CPT, Neurotron Inc., Baltimore, MD). It delivered sine-wave stimuli at 5 Hz applied via two gold electrodes placed on the medial and lateral side of the mid-phalanx of the right middle finger. Their intensity was increased from 0 to 20 mA by 0.2 mA/s until the subjects interrupted the current by releasing a button. Measurements were repeated five times every 30 s. Sensitization was assessed with punctate mechanical and heat11 stimuli and obtained using capsaicin cream (0.1 g, 0.1%, manufactured by the local pharmacy) applied onto a 3 × 3 cm skin area on the left volar forearm and covered with a plaster for 20 min. For cold stimuli12 a menthol solution (2 ml of a 40% menthol solution dissolved in ethanol) was used instead of capsaicin creme.
Data analysis
The goal of the data analysis was to (1) identify structure in the pain threshold data reflecting the prior classification of study participants by sex, (2) to determine which variables among pain threshold drive this structure, and (3) to translate this into a molecular hypothesis for sex-dependent pain perception and treatment. A variety of analytical techniques have been applied, including artificial intelligence (AI) methods selected from its currently most widely used subfield, machine learning. Overviews of the goals and applications of machine learning in pain research have been provided elsewhere10,13. In the present analyses, the focus was on detecting relevant structure in the data consistent with sex segregation. This was addressed using unsupervised and supervised methods. The main difference is the prior class labeling of the cases, i.e., in unsupervised analysis, the data structure is detected without knowledge of the class structure. In supervised methods, the algorithms are given training data, i.e., variables obtained from the cases, to learn the assignment of a case to a particular class. The success of this learning process is supervised because the class labels of the training samples are known. Only later, the trained algorithms have to prove their learned skill on new cases, using the same kind of variables they were trained with to assign a case to a particular class. Perhaps a school analogy will make this clearer: during learning, the “teacher/supervisor” (researcher/programmer) supervises the learning progress of the algorithms based on the knowledge of the correct result. Only when the task has been successfully learned can the algorithm be trusted to assign the correct class label (in this case, male/female) to new cases. These were available as a 20% validation sample, separated from the data before the algorithms were trained (see below). Both classes of analysis benefit from eliminating unnecessary information from relevant content, which is a standard part of machine learning workflow referred to as feature selection14. In the present analysis, several methods from both classes have been applied to obtain robust and internally validated results not relying on a single method.
The programming was performed in the R language15 using the R software package6, version 4.2.2 for Linux, available from the Comprehensive R Archive Network (CRAN) at https://CRAN.R-project.org/, and in the Python language16 using Python version 3.8.13 for Linux, available free of charge at https://www.python.org. Analyses were performed on 1–64 cores/threads on an AMD Ryzen Threadripper 3970X (Advanced Micro Devices, Inc., Santa Clara, CA, USA) computer running Ubuntu Linux 22.04.1 LTS (Canonical, London, UK), except that deep learning neural networks were trained on an NVIDIA GeForce RTX 3060 graphics processing unit (GPU) (NVIDIA Corporation, Santa Clara, CA, USA).
Data preprocessing
Data preprocessing consisted of log-transformation and imputation of censored data described in full detail in the supplementary materials. This provided a 125 × 11 sized input data space, X, of pain thresholds (125 subjects, 11 variables; Fig. 2), while the output data space, y, consisted of the binary sex information.
Unsupervised analyses
Unsupervised methods were applied to analyze the univariate modal distribution of variables and cluster structures in the multivariate data space. Unsupervised methods included Gaussian mixture modeling for univariate data, five different and projection methods for multivariate data (principal component analysis (PCA)17, independent component analysis (ICA)18, multidimensional scaling (MDS)19,20, isomap21 and t-distributed stochastic neighborhood embedding (t-SNE)22) and seven different clustering methods (k-means clustering23, partitioning around medoids (PAM)24, hierarchical clustering with Ward’s25, average, single, median and complete linkage). Details and results of the unsupervised analyses are reported in the supplementary materials.
Supervised analyses were performed in which machine learning algorithms were trained to infer the sex of the subjects from pain-related information. Supervised methods included five different machine learning algorithms (support vector machines (SVM)26, random forests27,28, logistic regression29, linear discriminant analysis (LDA)30 and a “deep learning”31 neural network32,33). Set-up and results of supervised analyses are described in detail as follows.
Supervised analyses
Supervised structure detection assumed that if a computational algorithm can be trained with pain threshold data to assign a subject to the correct class, i.e., sex, so that it can infer the sex of new subjects from their pain threshold pattern, then the pain thresholds contain information that is actually relevant to the sex of the subjects.
Supervised analyses were performed using cross-validation and training/test/validation splits of the data set. Specifically, prior to feature selection and classifier training, a class-proportional random sample of 20% of the dataset was set aside as a validation sample that was not touched further during algorithm training and feature selection. The 80/20% split of the data set was done in an automated manner using our R package “opdisDownsampling� (https://cran.r-project.org/package=opdisDownsampling), which selected from 106 random samples the one in which the distributions of the variables were most similar to those of these variables in the complete data set34. The remaining 80% of the dataset were considered as training sample, in which feature selection and classifier training were performed, using further training/test splits of this “training� sample in cross-validation settings; however, without touching the 20%-validation sample.
Feature selection
Feature selection aimed at filtering the “needles” from the “haystack”, i.e., identifying the relevant pain threshold variables for the sex differences. Several feature selection methods were used14 available in the Python package “scikit-learnâ€� (https://scikit-learn.org/stable/35). Univariate methods were used that are based on the false positive rate test and the family wise error rate. Multivariate feature selection methods were used that are based on calculation of an importance estimate for each variable following training of four different classification algorithms (SVM, random forests, logistic regression, LDA). After training of the algorithms, the most relevant variables were selected using methods from the “sklearn.feature_selection” module of scikit-learn. This included the F value based feature selection as implemented in the “SelectKBest” method. The number k of the features to be selected was determined by a grid search of [1,…,11] variables. Further methods comprised “SelectFromModel” (SFM), which selects features based on importance weights in the trained algorithm, recursive feature elimination (RFE), which selects features, by recursively considering smaller and smaller feature sets and generating a feature ranking, forward and backward sequential feature selection (SFS), which iteratively find the best features by adding features to a set of initially zero and all features, respectively. Furthermore, the generic permutation importance provided in the “permutation_importanceâ€� method of the “sklearn.inspectionâ€� package was used, setting the number of permutations to n_repeats = 50. The feature selection methods were applied in a 5 × 20 nested cross-validation scenario provided with the “RepeatedStratifiedKFold” method from the “sklearn.model_selection” module of “scikit-learn”, setting the parameters “n_splitsâ€� = 5 and “n_repeatsâ€� = 20. All classifiers used during feature selection were tuned with respect to relevant hyperparameters.
With the two univariate methods and the six multivariate methods run with four classification algorithms, the total number of feature selection methods summed to m = 26. In the resulting 11 × 26 feature matrix, i.e., 11 pain threshold variables versus 26 selection methods, initially filled with zeros, each feature received a value of 1 when selected. This resulted in a sum score for the number of selections for each pain-related variable. The final feature set was obtained from the majority vote of the 26 selection methods. Therefore, the sum score was submitted to computed ABC (cABC) analysis36, an item categorization technique that divides a set of positive numerical data into three disjoint subsets labeled “Aâ€� to “Câ€�. Subset “A” contains the “important few”, which were retained as “reducedâ€� feature sets, whereas subset Câ€� contains the “trivial manyâ€�37. The Python implementation is available as our package “cABCanalysisâ€� at https://pypi.org/project/cABCanalysis/38.
Classifier performance evaluation
The main classifiers were imported from the Python package “scikit-learnâ€� (https://scikit-learn.org/stable/35). SVM, implemented as “linearSVCâ€� and random forests were selected as two commonly used classification algorithms of different types, i.e., class separation using hyperplanes in data projected to higher dimensions, or class separation using an ensemble of simple decision trees. In addition, logistic regression and LDA were included as classical methods for class assignment. For comparison, a “deep learning”31 neural network32,33 was used to include a completely different type of classifier, using the GPU-only version of the “TensorFlowâ€� machine learning platform (https://www.tensorflow.org39) for Python.
The classifiers were tuned to the respective training data set using a grid search approach for different combinations of hyperparameters. For example, the tuning of the algorithms led to the selection of ridge regression as the regularization method for SVM and logistic regression, to a forest size of d = 500 trees with a maximum depth of 1 decision for random forests. For the deep neural network, after grid searches including the number of layers, number of neurons, several activation functions provided in “TensorFlow�, the number of hidden layers was set to h = 2, the number of neurons in layer 1 was set to int((nfeatures + 1)/0.5) and the number of neurons in layer 2 was set to int((nfeatures + 1)/1). A dropout layer with a rate of 0.2 was added after each layer to reduce possible overfitting. The activation functions were linear between the hidden layers and sigmoidal for the binary (men = 0, women = 1) output layer (Fig. 3). Binary crossentropy was set as the loss function, and 1000 epochs were used to train the neural network.
After feature selection, it was tested whether the selected pain threshold variables actually provided sufficient information for sex separation in a sample that was not available during classifier tuning and feature selection. Therefore, the included algorithms were trained with the full and reduced feature sets in a 5 × 20 nested cross-validation scenario, using randomly selected subsets of 67% of the original training dataset. The trained classifiers were then applied to random subsets comprising each 80% of the validation dataset that had been separated from the full original dataset prior to feature selection and classifier tuning. For the reduced feature sets, hyperparameter tuning was repeated prior to classifier performance evaluation. Balanced accuracy was used as the main parameter to evaluate the classification performance41. Additionally, the area under the receiver operating characteristics curve (roc-auc)42 was calculated. To control possible overfitting, all machine-learning algorithms were trained with pain data in that each variable was randomly permuted, with the expectation that a classifier trained with this information should not perform better than guessing, i.e., give a balanced accuracy (or a roc-auc) around 50%, else, overfitting could not be ruled out entirely. Furthermore, using a validated approach to the most informative features43, classifiers were also trained with the unselected features, i.e., the pain threshold variables that were kept as the result of feature selection. This provided further support that the feature selection had indeed identified the key informative variables.
Hypothesis transfer to the level of molecular pharmacology
From the machine-learning based analysis of the sex-related information contained in pain thresholds, a molecular hypothesis for sex-dependent pain perception and treatment could be derived that narrowed the molecular background on nociceptors for mechanically but not thermally induced pain. This was addressed via knowledge retrieval from publicly accessible databases. Relevant nociceptors were identified by domain expert knowledge and classical literature search among biomedical publications in the PubMed database. The evidence on the molecular background of pain induced by mechanical stimuli provided a basis for potential translation into precision pain medicine approaches.
To transfer this in a pharmacological context, information about drugs and their molecular targets was queried from the DrugBank database44 at https://go.drugbank.com (version 5.1.9 dated 2022-01-04). The database was downloaded as an extensible markup language (XML) file from https://go.drugbank.com/releases/5-1-9/downloads/all-full-database. The information contained in it was processed using the R package “dbparser� (https://cran.r-project.org/package=dbparser45). The drug targets were available encoded as Universal Protein Resource (UniProt) (https://www.uniprot.org46) IDs and converted into NCBI numbers of the coding genes using the R library “org.Hs.eg.db� (https://bioconductor.org/packages/release/data/annotation/html/org.Hs.eg.db.html47). Approved or investigational drugs with human targets were retained.
Ethics approval and consent to participate
The study from which the data set originates followed the Declaration of Helsinki and was approved by the Ethics Committee of Medical Faculty of the Goethe-University, Frankfurt am Main, Germany (approval number 150/11).
Informed consent
Permission for anonymized reports of data analysis results obtained from the acquired information was included in the informed written consent.
Results
Unsupervised analysis using five different common data projection methods and seven different common clustering algorithms ended up with a heterogeneous picture about the data structure and its correspondence to the prior classes. The respective results are reported in the supplemental information.
Sex separation achieved with machine-learning algorithm after supervised training
Pain threshold variables relevant to sex segregation
Among the 26 approaches to feature selection, pain threshold for blunt pressure stimuli was selected by a majority of 22 approaches (Fig. 4). Item categorization obtained by applying cABC analysis to the sum of votes per approach identified d = 4 variables as belonging to category “A�, i.e., the “important few� according to the classical nomenclature for ABC analysis37, namely pain thresholds to blunt pressure, punctate pressure with and without hypersensitization by capsaicin cream, and electrical current (Fig. 4). To further narrow the focus to the most relevant variables, a recursive cABC analysis was performed, which consists of repeating the cABC analysis for the items in category “A� of a precedent cABC analysis. This provided a “sparse� feature set comprising d = 1 variable, i.e., the pain threshold to blunt pressure (Fig. 4).
Ability of algorithms to identify a person’s sex based on pain sensitivity
When the algorithms were trained with the full pain threshold information (d = 11 variables) from 80% of the study subjects, they identified sex in the other 20% of the subjects with a median balanced accuracy of up to 79% (Table 1). This remained almost unchanged when the algorithms were trained with the d = 4 pain thresholds of the “reduced� feature set. Even with only d = 1 pain threshold of the “sparse� feature set, i.e., pain thresholds to blunt pressure, the balanced accuracy was up to 79%.
By contrast, when training the algorithms with the unselected pain thresholds, to which belonged the thermal pain thresholds and the effects of sensitization, they could not identify the subjects’ sex better than by guessing. Classification performance was then equivalent to that obtained when permuted data, i.e., nonsense information, was used for training (Fig. 4). This underscores that the informative pain threshold information in the sex context had been indeed identified during feature selection (Fig. 4).
Possible molecular-pharmacological implications
Evidence identified by literature search for nociceptors involved in the perception of pain elicited by mechanical but not thermal noxious stimuli, nor by the hypersensitization induced by capsaicin and menthol, pointed to d = 13 molecular targets (Table 2). Information retrieval from the DrugBank database provided a 79 × 13 matrix of drugs versus nociceptive targets (Fig. 5). Most of the 79 drugs were calcium channel blockers targeting cardiovascular diseases and might be of interest for drug repurposing to target pain. Of the in total 14,594 unique entries in the DrugBank database, 229 drugs were annotated with a therapeutic indication that contained the substrings “pain� or “nocicep� or “analge�. Of those, 10 were among the 79 drugs in above matrix, namely tramadol, enflurane, methadone, chlorzoxazone, meperidine, levomenthol, ethanol, orphenadrine, cns-5161, ketobemidone. Most are well-known substances. Of the lesser-known ones, orphenadrine is mentioned in the DrugBank database as a muscarinic antagonist used to treat drug-induced parkinsonism and to relieve pain from muscle spasm, and CNS-5161 is a N-methyl-d-aspartate ion channel antagonist.
Discussion
The application of machine learning principles59 allowed the inversion of the association of sex and pain sensitivity. A median balanced accuracy of 74–79% can be considered as good according to the usual criteria for machine learning classifiers. This strongly indicates that a person’s sex is highly relevant in pain research and probably in pain treatment. The analyses showed that sex segregation is not evenly distributed across the pain thresholds to different noxious stimuli. Positive evidence was provided that sex can be inferred from mechanical thresholds and negative evidence that it cannot be inferred from thermal thresholds or from the effects of capsaicin/menthol induced hypersensitization. The “not” logic, based on consideration of the unselected features43, is likely to hold as well to other machine learning algorithms than the presently used selection, which already covers various architectures up to “deep learning”. None of the algorithms showed any tendency to leave the guessing range when trained with the unselected features, i.e., thermal pain thresholds or sensitization effects (Fig. 6).
Implications for pain precision medicine
The logic of “mechanical but not thermal” can be directly translated to the molecular level by focusing on pathways. Three main types of nociceptors are expressed in the skin: mechanosensitive and mechanothermal nociceptors, which signal mainly through myelinated Aδ-nerve fibers, and polymodal nociceptors, which are connected to unmyelinated C-nerve fibers60. Mechanical and thermal stimuli elicit pain through different pathways; for example, neurons activated by the peripheral application of heat differ from those activated by noxious pressure61. Mechanoreceptors and mechano-transducers include ion channels, specialized cytoskeletal proteins, cell junction molecules, and G protein-coupled receptors. There is growing evidence that mechanical stimuli play a role in pathological pain conditions62,63,64.
Ion channels that have been shown to be involved in the perception of mechanical stimuli include the so-called piezo-receptors, PIEZO1 and PIEZO2, which are currently the largest known ion channels with three identical subunits of approximately 2500 amino acids each and are highly conserved in vertebrates65. These ion channels control the uptake of cationic ions, particularly Ca2+, in response to a stimulus, allowing them to mediate action potentials and intracellular signal transduction through second messenger signaling66. Ca2+ entry mediated by mechanosensitive ion channels is also tightly regulated by a variety of key proteins distributed in the cell membrane or endoplasmic reticulum. Recent studies have shown that mechanosensitive ion channels can even physically interact with Ca2+-regulatory proteins, and these interactions have far-reaching implications for physiology and pathophysiology67.
The frequency of action potentials in nociceptors is a major determinant of pain intensity. Modulators include HCN ion channels, which generate an inward current following membrane hyperpolarization68. The human HCN2 channel is activated by hyperpolarization that exhibits weak selectivity for potassium versus sodium ions69. It contributes to native pacemaker currents in the heart and neurons. These channels have recently been linked to chronic pain. Blocking the HCN2 gene has been associated with the disappearance of neuropathic pain episodes70. In addition, HCN2 has been shown to interact with HCN171 and HCN471 by co-assembling homologous subunits to heteromeric complexes.
One possible mechanism for the sex-specificity of mechanical nociceptors could be an allosteric interaction with estrogens. That is, while receptors are physiologically stimulated by endogenous orthosteric agonists, few naturally occurring allosteric modulators have been described72. Steroid hormones such as progesterone and estradiol exert allosteric effects on cannabinoid73, oxytocin74, and glycoprotein hormone receptors75, respectively, implying that receptors can be indirectly modulated by changing the concentration of the endogenous allosteric modulator76. Recently, neurosteroids and neuroactive steroids were shown to act as allosteric modulators of muscarinic receptors, which are a subfamily of G-protein receptors77,78 and GABA receptors, a class of receptors that respond to the neurotransmitter gamma-aminobutyric acid (GABA), the major inhibitory compound in the mature vertebrate central nervous system79. However, the relevance of these allosteric mechanisms to sex-specific mechanical pain remains to be demonstrated, as corresponding evidence for mechanical nociceptors is still lacking in the literature.
Pharmacological application of allosteric modulation is conceivable with emerging approaches to drug design based on secondary binding site effects, where small molecule drugs are designed to bind into secondary binding sites on the targeted biomolecules80. According to the present line of reasoning, drugs addressing the molecular targets of mechanical nociceptive stimuli might act better on pain in women than in men and thus be suitable for sex-specific analgesic treatment. For the 10 drugs annotated with effects on pain, evidence for a better effect in women is sparse81. Nonetheless, the present explorative analysis illustrates the possibility that the data analysis approach presented in this report could be used for sex-specific precision medicine in pain. Certain analgesics could be preferred or future approaches to drug repurposing could be guided. Of course, this would also require the identification of clinical pain settings in which mechanical stimuli play a main role. To date, sex specific analgesia has been rarely implemented clinically. However, proof-of-concept is provided through the observation that κ-opioid agonists (pentazocine) are more efficient in women with the “red hair—white skin” phenotype based on variants of the melanocortin-1 receptor gene (MC1R)82.
While it seems accepted that females have lower pain thresholds for various forms of noxious stimuli83, the mechanistic basis of this sexual dimorphism in nociception is far from being fully understood. Transcriptome analyses of the whole dorsal root ganglion of female rats have revealed lower cutaneous mechanical nociceptive thresholds84. Several studies have shown that in antinociceptive procedures using thermal, chemical and electrical stimuli, opioids are generally more potent in male than in female rodents, and vice versa85. It has also been shown that temporal summation of heat pain is greater in females than in males, suggesting that central processing of nociceptive input may be upregulated in females86. Several further studies have investigated temporal summation to noxious mechanical stimulation and examined sex differences in temporal summation of mechanically evoked pain87. Their results indicate that temporal summation of mechanically evoked pain is higher in females than in males, is stimulus frequency dependent, and is centrally mediated86.
The absence of ovarian hormones caused an increase in mean arterial pressure and a decrease in paw nociception in female rodents. Ovarian hormones play an important role in the mechanical threshold. The ovaries secrete two important hormones, estrogen and progesterone, which play a role in pain modulation88. Furthermore, many chronic pain conditions show sex differences in their epidemiology89. This may be due to sex-specific differential expression of genes involved in nociceptive pathways, including immune cells, sensory neurons, and the neuroendocrine system90. Elucidating how different cell types detect and respond to noxious stimuli may lead to new therapeutic insights and a better understanding of the basic mechanisms of pain plasticity.
Implications for pain data analysis
Seeking structure in data sets can be complex and the standard approach of perform a principal component analysis (PCA) to project data followed by group segregation statistics may not suffice always. In the present data set, if any structure and clusters could be detected in the data with unsupervised analyses (for a complete description of methods and results of unsupervised analyses, see Supplementary information), then after t-SNE (t-distributed stochastic neighborhood embedding) projection (Supplementary Fig. 5). Supervised approaches to structures that reflected the sex structure in the present data provided more convincing results. However, it was not possible to determine with certainty in advance which method was successful in the present data set. This emphasizes that a complete analysis of pain-related data structures should include several methods.
Second, by using a machine-learning approach, the research question of sex differences in pain could be reversed by addressing the question whether pain can be used to infer sex. This allowed address negative evidence with the thermal pain thresholds directly. That is, the absence of positive evidence as provided in statistically non-significant sex differences in thermal pain thresholds could be extended to a presence of negative evidence by showing that sex inference from thermal thresholds was not better than guessing, similar to training the algorithms with “nonsense� information created by permutation of the variables. This emphasizes the need for negative evidence, regarded as evidence for non-involvement and not only as absent evidence for involvement, which is rarely provided in the scientific literature. For the present molecular targets (Table 2), explicit evidence for non-involvement in thermal pain was found only for HCN291.
Finally, it should be noted that the present analyses cannot cover the full possible spectrum of AI and machine learning in pain research. While several different methods of data analysis were implemented, it is not possible to cover all (for an overview, see58). Similarly, the deep learning neural network32,33 was trained only in one of the discriminative forms (for an overview, see e.g.59) while further architectures were not tested. However, without including other architectures, the utility of deep learning for datasets such as the present one cannot be conclusively judged.
Conclusions
Formulating the question of sex differences in pain as a machine learning problem allowed for an unbiased analysis that concluded that pain thresholds differ relevantly, to the extent that algorithms can be trained with pain threshold information from men and women that they are able to correctly infer sex in new subjects by up to 79% balanced accuracy. This underscores that sex cannot be ignored in pain research and probably in analgesic therapy. By addressing a subsequent hypothesis that included a “NOT” logic, machine learning indicated that sex-relevant information resides in the pain thresholds for mechanical but not thermal stimuli of responses to local hypersensitization. The hypothesis thus formulated could be translated into a molecular hypothesis about nociceptive targets. The present analytic approach highlights that a diverse composition of data science methods including machine learning algorithms can help decipher deep data structures relevant to biomedical and clinical professionals.
Data availability
Data are available from the first author on request.
References
-
Fillingim, R. B., King, C. D., Ribeiro-Dasilva, M. C., Rahim-Williams, B. & Riley, J. L. 3rd. Sex, gender, and pain: A review of recent clinical and experimental findings. J. Pain 10, 447–485. https://doi.org/10.1016/j.jpain.2008.12.001 (2009).
-
Derbyshire, S. W. Gender, pain, and the brain. Pain Clin. Updates 20, 20 (2008).
-
Unruh, A. M. Gender variations in clinical pain experience. Pain 65, 123–167 (1996).
-
Petrovich, D. V. The pain apperception test: An application to sex differences. J. Clin. Psychol. 15, 412–414. https://doi.org/10.1002/1097-4679(195910)15:4%3c412::aid-jclp2270150413%3e3.0.co;2-c (1959).
-
Kovalchik, S. RISmed: Download Content from NCBI Databases. R package version 2.3.0, https://CRAN.R-project.org/package=RISmed (2021).
-
R Development Core Team. R: A Language and Environment for Statistical Computing (R Development Core Team, 2008).
-
Wickham, H. ggplot2: Elegant Graphics for Data Analysis (Springer, 2009). https://cran.r-project.org/package=ggplot2.
-
Samuel, A. L. Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 3, 210–229. https://doi.org/10.1147/rd.33.0210 (1959).
-
Doehring, A. et al. Effect sizes in experimental pain produced by gender, genetic variants and sensitization procedures. PLoSÂ One 6, e17724. https://doi.org/10.1371/journal.pone.0017724 (2011).
-
Lötsch, J., Ultsch, A., Mayer, B. & Kringel, D. Artificial intelligence and machine learning in pain research: A data scientometric analysis. Pain Rep. 7, 25 (2022).
-
Petersen, K. L. & Rowbotham, M. C. A new human experimental pain model: The heat/capsaicin sensitization model. NeuroReport 10, 1511–1516 (1999).
-
Hatem, S., Attal, N., Willer, J. C. & Bouhassira, D. Psychophysical study of the effects of topical application of menthol in healthy volunteers. Pain 122, 190–196 (2006).
-
Lotsch, J. & Ultsch, A. Machine learning in pain research. Pain 159, 623–630. https://doi.org/10.1097/j.pain.0000000000001118 (2017).
-
Guyon, I. An introduction to variable and feature selection. J. Mach. Learn. Res. 3, 1157–1182 (2003).
-
Ihaka, R. & Gentleman, R. R: A language for data analysis and graphics. J. Comput. Graph. Stat. 5, 299–314. https://doi.org/10.1080/10618600.1996.10474713 (1996).
-
Van Rossum, G. & Drake, F. L. Jr. Python Tutorial Vol 620 (Centrum voor Wiskunde en Informatica, 1995).
-
Pearson, K. L. I. I. I. On lines and planes of closest fit to systems of points in space. Lond. Edinb. Dublin Philos. Mag. J. Sci. 2, 559–572. https://doi.org/10.1080/14786440109462720 (1901).
-
Hyvärinen, A. & Oja, E. Independent component analysis: Algorithms and applications. Neural Netw. 13, 411–430. https://doi.org/10.1016/S0893-6080(00)00026-5 (2000).
-
Shepard, R. N. The analysis of proximities: Multidimensional scaling with an unknown distance function. II. Psychometrika 27, 219–246 (1962).
-
Shepard, R. N. The analysis of proximities: Multidimensional scaling with an unknown distance function. I. Psychometrika 27, 125–140 (1962).
-
Tenenbaum, J. B., de Silva, V. & Langford, J. C. A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323. https://doi.org/10.1126/science.290.5500.2319 (2000).
-
Van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).
-
MacQueen, J. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics. 281–297 (University of California Press).
-
Kaufman, L. & Rousseeuw, P. J. Partitioning around medoids (program PAM). Finding Groups Data https://doi.org/10.1002/9780470316801.ch2 (1990).
-
Ward, J. H. Jr. Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58, 236–244 (1963).
-
Cortes, C. & Vapnik, V. Support-vector networks. Mach. Learn. 20, 273–297. https://doi.org/10.1023/a:1022627411411 (1995).
-
Ho, T. K. In Proceedings of the Third International Conference on Document Analysis and Recognition (Volume 1)—Volume 1 278 (IEEE Computer Society, 1995).
-
Breiman, L. Random forests. Mach. Learn. 45, 5–32. https://doi.org/10.1023/a:1010933404324 (2001).
-
Cramer, J. S. The Origins of Logistic Regression (Tinbergen Institute, 2002).
-
Fisher, R. A. The use of multiple measurements in taxonomic problems. Ann. Eugen. 7, 179–188. https://doi.org/10.1111/j.1469-1809.1936.tb02137.x (1936).
-
Deng, L. & Yu, D. Deep learning: Methods and applications. Found. Trends Signal Process. 7, 197–387 (2014).
-
McCulloch, W. S. & Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5, 115–133 (1943).
-
Hinton, G. E., Osindero, S. & Teh, Y.-W. A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006).
-
Lötsch, J., Malkusch, S. & Ultsch, A. Optimal distribution-preserving downsampling of large biomedical data sets (opdisDownsampling). PLoS One 16, e0255838. https://doi.org/10.1371/journal.pone.0255838 (2021).
-
Pedregosa, F. et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
-
Ultsch, A. & Lötsch, J. Computed ABC analysis for rational selection of most informative variables in multivariate data. PLoS One 10, e0129767. https://doi.org/10.1371/journal.pone.0129767 (2015).
-
Juran, J. M. The non-Pareto principle; Mea culpa. Qual. Progress 8, 8–9 (1975).
-
Lötsch, J. & Ultsch, A. Recursive computed ABC (cABC) analysis as a precise method for reducing machine learning based feature sets to their minimum informative size. Sci. Rep. 13(1), 5470. https://doi.org/10.1038/s41598-023-32396-9 (2023).
-
Abadi, M. n. et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. CoRR. vol. abs/1603.04467. http://arxiv.org/abs/1603.04467 (2016).
-
Waskom, M. L. seaborn: Statistical data visualization. J. Open Source Softw. 6, 3021 (2021).
-
Brodersen, K. H., Ong, C. S., Stephan, K. E. & Buhmann, J. M. In Pattern Recognition (ICPR), 2010 20th International Conference on. 3121–3124.
-
Peterson, W., Birdsall, T. & Fox, W. The theory of signal detectability. Trans. IRE Prof. Group Inf. Theory 4, 171–212. https://doi.org/10.1109/TIT.1954.1057460 (1954).
-
Lötsch, J. & Ultsch, A. Enhancing explainable machine learning by reconsidering initially unselected items in feature selection for classification. BioMedInformatics 2, 701–714 (2022).
-
Wishart, D. S. et al. DrugBank 5.0: A major update to the DrugBank database for 2018. Nucleic Acids Res. 46, D1074-d1082. https://doi.org/10.1093/nar/gkx1037 (2018).
-
Ali, M. & Ezzat, A. dbparser: DrugBank Database XML Parser. R package version 1.2.0. https://cran.r-project.org/package=dbparser (2020).
-
UniProt: The universal protein knowledgebase in 2021. Nucleic Acids Res. 49, D480–D489. https://doi.org/10.1093/nar/gkaa1100 (2021).
-
Carlson, M. org.Hs.eg.db: Genome wide annotation for Human. R package version 3.16.0. https://bioconductor.org/packages/release/data/annotation/html/org.Hs.eg.db.html (2020).
-
Ruan, N. et al. Acid-sensing ion channels and mechanosensation. Int. J. Mol. Sci. https://doi.org/10.3390/ijms22094810 (2021).
-
Efremov, A. K. et al. Application of piconewton forces to individual filopodia reveals mechanosensory role of L-type Ca2+ channels. Biomaterials 284, 121477. https://doi.org/10.1016/j.biomaterials.2022.121477 (2022).
-
Ventéo, S. et al. Fxyd2 regulates Aδ- and C-fiber mechanosensitivity and is required for the maintenance of neuropathic pain. Sci. Rep. 6, 36407. https://doi.org/10.1038/srep36407 (2016).
-
Paoletti, P. & Ascher, P. Mechanosensitivity of NMDA receptors in cultured mouse central neurons. Neuron 13, 645–655. https://doi.org/10.1016/0896-6273(94)90032-9 (1994).
-
Emery, E. C., Young, G. T. & McNaughton, P. A. HCN2 ion channels: An emerging role as the pacemakers of pain. Trends Pharmacol. Sci. 33, 456–463. https://doi.org/10.1016/j.tips.2012.04.004 (2012).
-
Al-Shammari, H. et al. Expression and function of mechanosensitive ion channels in human valve interstitial cells. PLoSÂ One 15, e0240532. https://doi.org/10.1371/journal.pone.0240532 (2020).
-
Kauffenstein, G. et al. Central role of P2Y6 UDP receptor in arteriolar myogenic tone. Arterioscler. Thromb. Vasc. Biol. 36, 1598–1606. https://doi.org/10.1161/atvbaha.116.307739 (2016).
-
Coste, B. et al. Piezo proteins are pore-forming subunits of mechanically activated channels. Nature 483, 176–181. https://doi.org/10.1038/nature10812 (2012).
-
Rossier, B. C. Mechanosensitivity of the epithelial sodium channel (ENaC): Controversy or pseudocontroversy?. J. Gen. Physiol. 112, 95–96. https://doi.org/10.1085/jgp.112.2.95 (1998).
-
Geng, Y. & Magleby, K. L. Single-channel kinetics of BK (Slo1) channels. Front. Physiol. 5, 532. https://doi.org/10.3389/fphys.2014.00532 (2014).
-
Gu, Z., Eils, R. & Schlesner, M. Complex heatmaps reveal patterns and correlations in multidimensional genomic data. Bioinformatics 32, 2847–2849. https://doi.org/10.1093/bioinformatics/btw313 (2016).
-
Murphy, K. P. Machine Learning: A Probabilistic Perspective (The MIT Press, 2012).
-
Eilers, H. & Schumacher, M. A. In Mechanosensitivity in Cells and Tissues (eds Kamkin, A. & Kiseleva, I.) (Academia Publishing House Ltd., 2005).
-
Lariviere, W. R. et al. Heritability of nociception. III. Genetic relationships among commonly used assays of nociception and hypersensitivity. Pain 97, 75–86. https://doi.org/10.1016/s0304-3959(01)00492-4 (2002).
-
Lolignier, S., Eijkelkamp, N. & Wood, J. N. Mechanical allodynia. Pflugers Arch. 467, 133–139. https://doi.org/10.1007/s00424-014-1532-0 (2015).
-
Kuner, R. Central mechanisms of pathological pain. Nat. Med. 16, 1258–1266. https://doi.org/10.1038/nm.2231 (2010).
-
Coderre, T. J., Katz, J., Vaccarino, A. L. & Melzack, R. Contribution of central neuroplasticity to pathological pain: Review of clinical and experimental evidence. Pain 52, 259–285 (1993).
-
Zarychanski, R. et al. Mutations in the mechanotransduction protein PIEZO1 are associated with hereditary xerocytosis. Blood 120, 1908–1915. https://doi.org/10.1182/blood-2012-04-422253 (2012).
-
Chubinskiy-Nadezhdin, V. I. et al. Local calcium signalling is mediated by mechanosensitive ion channels in mesenchymal stem cells. Biochem. Biophys. Res. Commun. 482, 563–568. https://doi.org/10.1016/j.bbrc.2016.11.074 (2017).
-
Wang, Y., Shi, J. & Tong, X. Cross-talk between mechanosensitive ion channels and calcium regulatory proteins in cardiovascular health and disease. Int. J. Mol. Sci. https://doi.org/10.3390/ijms22168782 (2021).
-
Emery, E. C., Young, G. T., Berrocoso, E. M., Chen, L. & McNaughton, P. A. HCN2 ion channels play a central role in inflammatory and neuropathic pain. Science 333, 1462–1466. https://doi.org/10.1126/science.1206243 (2011).
-
Vaccari, T. et al. The human gene coding for HCN2, a pacemaker channel of the heart. Biochim. Biophys. Acta 1446, 419–425. https://doi.org/10.1016/s0167-4781(99)00092-5 (1999).
-
Lainez, S., Tsantoulas, C., Biel, M. & McNaughton, P. A. HCN3 ion channels: Roles in sensory neuronal excitability and pain. J. Physiol. 597, 4661–4675. https://doi.org/10.1113/jp278211 (2019).
-
Proenza, C. et al. Different roles for the cyclic nucleotide binding domain and amino terminus in assembly and expression of hyperpolarization-activated, cyclic nucleotide-gated channels. J. Biol. Chem. 277, 29634–29642. https://doi.org/10.1074/jbc.M200504200 (2002).
-
Melancon, B. J. et al. Allosteric modulation of seven transmembrane spanning receptors: Theory, practice, and opportunities for central nervous system drug discovery. J. Med. Chem. 55, 1445–1464. https://doi.org/10.1021/jm201139r (2012).
-
Morales, P., Goya, P., Jagerovic, N. & Hernandez-Folgado, L. Allosteric modulators of the CB(1) cannabinoid receptor: A structural update review. Cannabis Cannabinoid. Res. 1, 22–30. https://doi.org/10.1089/can.2015.0005 (2016).
-
Grazzini, E., Guillon, G., Mouillac, B. & Zingg, H. H. Inhibition of oxytocin receptor function by direct binding of progesterone. Nature 392, 509–512. https://doi.org/10.1038/33176 (1998).
-
Rossi, M. et al. Presence of a putative steroidal allosteric site on glycoprotein hormone receptors. Eur. J. Pharmacol. 623, 155–159. https://doi.org/10.1016/j.ejphar.2009.09.029 (2009).
-
Fasciani, I. et al. Allosteric modulators of G protein-coupled dopamine and serotonin receptors: A new class of atypical antipsychotics. Pharmaceuticals (Basel) https://doi.org/10.3390/ph13110388 (2020).
-
DolejÅ¡Ã, E. et al. Neurosteroids and steroid hormones are allosteric modulators of muscarinic receptors. Neuropharmacology 199, 108798. https://doi.org/10.1016/j.neuropharm.2021.108798 (2021).
-
Szczurowska, E., Szánti-Pintér, E., Randáková, A., JakubÃk, J. & Kudova, E. Allosteric modulation of muscarinic receptors by cholesterol, neurosteroids and neuroactive steroids. Int. J. Mol. Sci. https://doi.org/10.3390/ijms232113075 (2022).
-
Martinez Botella, G. et al. Neuroactive steroids. 1. Positive allosteric modulators of the (γ-aminobutyric acid)A receptor: Structure–activity relationships of heterocyclic substitution at C-21. J. Med. Chem. 58, 3500–3511. https://doi.org/10.1021/acs.jmedchem.5b00032 (2015).
-
Abdel-Magid, A. F. Allosteric modulators: An emerging concept in drug discovery. ACS Med. Chem. Lett. 6, 104–107. https://doi.org/10.1021/ml5005365 (2015).
-
Rauck, R. L. et al. A randomized, double-blind, placebo-controlled study of intrathecal ziconotide in adults with severe chronic pain. J. Pain Symptom Manage. 31, 393–406. https://doi.org/10.1016/j.jpainsymman.2005.10.003 (2006).
-
Mogil, J. S. et al. The melanocortin-1 receptor gene mediates female-specific mechanisms of analgesia in mice and humans. Proc. Natl. Acad. Sci. USA 100, 4867–4872 (2003).
-
Bartley, E. J. & Fillingim, R. B. Sex differences in pain: A brief review of clinical and experimental findings. Br. J. Anaesth. 111, 52–58. https://doi.org/10.1093/bja/aet127 (2013).
-
Hendrich, J. et al. In vivo and in vitro comparison of female and male nociceptors. J. Pain 13, 1224–1231. https://doi.org/10.1016/j.jpain.2012.09.009 (2012).
-
Barrett, A. C., Smith, E. S. & Picker, M. J. Sex-related differences in mechanical nociception and antinociception produced by mu- and kappa-opioid receptor agonists in rats. Eur. J. Pharmacol. 452, 163–173. https://doi.org/10.1016/s0014-2999(02)02274-4 (2002).
-
Sarlani, E. & Greenspan, J. D. Gender differences in temporal summation of mechanically evoked pain. Pain 97, 163–169. https://doi.org/10.1016/s0304-3959(02)00015-5 (2002).
-
Riley, J. L. 3rd., Robinson, M. E., Wise, E. A., Myers, C. D. & Fillingim, R. B. Sex differences in the perception of noxious experimental stimuli: A meta-analysis. Pain 74, 181–187 (1998).
-
Chen, Q., Zhang, W., Sadana, N. & Chen, X. Estrogen receptors in pain modulation: Cellular signaling. Biol. Sex Differ. 12, 22. https://doi.org/10.1186/s13293-021-00364-5 (2021).
-
Mecklenburg, J. et al. Transcriptomic sex differences in sensory neuronal populations of mice. Sci. Rep. 10, 15278. https://doi.org/10.1038/s41598-020-72285-z (2020).
-
Paige, C. et al. Neuroendocrine mechanisms governing sex differences in hyperalgesic priming involve prolactin receptor sensory neuron signaling. J. Neurosci. 40, 7080–7090. https://doi.org/10.1523/jneurosci.1499-20.2020 (2020).
-
Schnorr, S. et al. HCN2 channels account for mechanical (but not heat) hyperalgesia during long-standing inflammation. Pain 155, 1079–1090. https://doi.org/10.1016/j.pain.2014.02.006 (2014).
Acknowledgements
The pain data used in the present analyses had been analyzed in a different and non-redundant manner previously and were originally reported in PLoS One 6, e17724. https://doi.org/10.1371/journal.pone.0017724. In addition, from the same project but not covering the complete data set presented here, further analyses were published in Clin J Pain 25, 128–131. https://doi.org/10.1097/AJP.0b013e3181817aa2 and Pain 138, 286–291. https://doi.org/10.1016/j.pain.2007.12.015. Neither the present analyses nor their scientific targets are redundant to these previous works.
Funding
Open Access funding enabled and organized by Projekt DEAL. JL was supported by the Deutsche Forschungsgemeinschaft (DFG LO 612/16-1).
Author information
Authors and Affiliations
Contributions
J.L. conceptualization of the project, programming, writing of the manuscript, data analyses and creation of the figures. B.M. critical revision of the manuscript for important intellectual content, writing of the manuscript. D.K. translation of the findings to the molecular level, writing of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Lötsch, J., Mayer, B. & Kringel, D. Machine learning analysis predicts a person’s sex based on mechanical but not thermal pain thresholds.
Sci Rep 13, 7332 (2023). https://doi.org/10.1038/s41598-023-33337-2
-
Received:
-
Accepted:
-
Published:
-
DOI: https://doi.org/10.1038/s41598-023-33337-2
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.
Post Disclaimer
The information provided in our posts or blogs are for educational and informative purposes only. We do not guarantee the accuracy, completeness or suitability of the information. We do not provide financial or investment advice. Readers should always seek professional advice before making any financial or investment decisions based on the information provided in our content. We will not be held responsible for any losses, damages or consequences that may arise from relying on the information provided in our content.