Critical Assessment of Fully Automated Structure Prediction


EVALUATION OF CAFASP-DP METHODS


Number of targets evaluated:58
Number of single-domain targets:41
Number of two-domain targets:17

Due to the complex domain topology, the following five targets have not been currently evaluated:
T0226, T0248, T0268, T0279, T0280


The performance of domain prediction methods has been carried out based on following measures:

Separate values are computed for the single-domain targets, the two-domain targets and all targets.

Twelve CAFASP-DP methods have been evaluated along with InterProScan:

  1. Adda
  2. Armadillo
  3. Biozon
  4. Dompred-domssea
  5. Dompred-dps
  6. Dompro
  7. Dopro
  8. Globplot
  9. InterProScan
  10. Mateo
  11. Ssep-Domain
  12. Robetta-ginzu
  13. Robetta-rosettadom


  1. Absolute Number of Correctly Predicted Targets
  2. In addition to the 12 domain prediction servers, three controls have been used: Control1, Control2 and Random. Control1 is computed by predicting all the targets as single-domain proteins and control2 is computed by predicting all the targets as two-domain proteins with a domain boundary at the centre of the sequence. The random control chooses between control1 and control2 at random. Finally, a consensus has been calculated based on a majority vote and some weighting scheme in case of a tie.

    A prediction is considered as correct if the number of predicted domains is correct. For two-domain targets, the predicted domains should be continuous, i.e, with no split domains.

    MethodsNumber of targets predicted as single-domainNumber of targets predicted correctly as single-domainNumber of targets predicted as two-domainNumber of targets predicted correctly as two-domainNumber of targets predicted as multi-domain (number of domains >2) or split-domainsMissing Predictions
    Control158410000
    Control200581700
    Random302128800
    ADDA48359310
    Armadillo44224257
    Biozon44316221
    Dompred-Domssea44338515
    Dompred-DPS362816815
    Dompro463512600
    Dopro403514940
    Globplot48345314
    Interproscan51386410
    Mateo2721132180
    Ssep-domain453811820
    Robetta-Ginzu363313990
    Robetta-Rosettadom3634161260
    CONSENSUS4237141120

  3. Sensitivity and Specificity for single, two-domain and all targets
  4. For each method, sensitivity (Sen) and specificity (Spec) have been calculated and plotted separately for single-domain targets, two-domain targets and finally for all targets as

    Sen = TP / TP+FN

    Spec = TP / TP+FP

    where TP: Number of true positives; FN: Number of false negatives and FP: Number of false positives

  5. Overlap Score (Single-Domain Targets)
  6. Targets Control1 Control2 Random Adda Armadillo Biozon Dompred-domssea Dompred-dps Dompro Dopro Globplot Interproscan Mateo Ssep-domain Robetta-ginzu Robetta-rosettadom Consensus
    T0196 100.00 50.00 100.00 92.24 50.86 56.90 100.00 100.00 100.00 74.13 98.28 75.86 93.97 74.14 100.00 100.00 100.00
    T0197 100.00 50.28 50.28 94.97 70.39 39.11 100.00 52.51 100.00 93.30 98.32 92.18 96.09 92.74 60.34 59.78 100.00
    T0198 100.00 50.21 100.00 100.00 X 48.09 100.00 54.47 71.49 87.23 97.87 37.87 97.02 100.00 53.62 100.00 100.00
    T0200 100.00 50.20 50.20 92.16 X 34.90 100.00 100.00 100.00 100.00 34.12 69.41 70.59 100.00 100.00 100.00 100.00
    T0201 100.00 50.00 100.00 100.00 X 71.28 100.00 100.00 100.00 87.23 94.68 95.75 31.92 100.00 100.00 100.00 100.00
    T0203 100.00 50.00 50.00 99.48 23.04 45.55 100.00 100.00 100.00 100.00 40.31 94.24 98.17 100.00 65.71 65.71 100.00
    T0204 100.00 50.14 100.00 98.86 30.20 27.64 60.40 61.25 84.05 52.14 80.34 96.87 41.88 56.13 55.84 55.84 56.98
    T0205 100.00 50.00 50.00 83.85 100.00 74.62 100.00 100.00 100.00 67.69 98.46 53.85 93.85 100.00 100.00 100.00 100.00
    T0206 100.00 50.00 100.00 64.55 23.64 65.91 100.00 65.46 100.00 77.73 X 58.18 81.82 68.64 61.36 61.36 50.91
    T0208 100.00 50.14 50.14 48.74 57.42 58.82 100.00 62.19 54.34 96.08 56.58 98.60 77.87 97.48 75.63 75.63 61.35
    T0211 100.00 50.14 100.00 100.00 68.06 54.86 100.00 100.00 100.00 95.83 70.14 85.42 95.14 95.83 100.00 100.00 100.00
    T0212 100.00 50.00 50.00 96.03 52.38 59.52 100.00 100.00 100.00 93.65 X 96.03 94.44 96.03 100.00 100.00 100.00
    T0213 100.00 50.49 100.00 97.09 51.46 100.00 100.00 100.00 100.00 93.20 100.00 99.03 100.00 97.09 100.00 100.00 100.00
    T0214 100.00 50.00 50.00 95.46 58.18 65.46 100.00 100.00 100.00 98.18 100.00 91.82 100.00 92.73 100.00 100.00 100.00
    T0215 100.00 50.00 100.00 71.05 X 56.58 X X 100.00 100.00 X 71.05 50.00 100.00 100.00 100.00 100.00
    T0224 100.00 50.58 50.58 100.00 100.00 50.58 X X 100.00 100.00 97.70 100.00 49.43 100.00 100.00 100.00 100.00
    T0227 100.00 50.41 100.00 90.91 54.55 100.00 100.00 100.00 100.00 96.69 100.00 80.99 65.29 100.00 100.00 100.00 100.00
    T0230 100.00 50.00 50.00 95.19 49.04 73.08 100.00 100.00 100.00 100.00 99.04 73.08 93.27 100.00 100.00 100.00 100.00
    T0231 100.00 50.00 100.00 99.30 59.16 54.93 100.00 100.00 100.00 93.66 97.88 90.14 100.00 93.66 100.00 100.00 100.00
    T0234 100.00 50.30 50.30 100.00 56.97 67.27 100.00 100.00 100.00 100.00 98.18 56.97 42.42 100.00 100.00 100.00 100.00
    T0238 100.00 50.20 100.00 51.79 76.89 100.00 55.38 100.00 78.09 83.67 75.70 35.06 55.78 90.44 100.00 100.00 100.00
    T0239 100.00 50.00 50.00 100.00 53.06 64.29 X X 100.00 86.74 97.96 82.65 53.06 100.00 100.00 100.00 100.00
    T0240 100.00 50.00 100.00 84.44 48.89 100.00 X X 100.00 81.11 96.67 38.89 50.00 100.00 100.00 100.00 100.00
    T0241 100.00 50.21 50.21 100.00 34.18 47.26 53.17 100.00 100.00 62.45 100.00 94.52 29.96 56.12 70.04 45.99 45.99
    T0242 100.00 50.00 100.00 99.14 49.14 74.14 100.00 100.00 100.00 58.62 99.14 99.14 59.48 100.00 100.00 100.00 100.00
    T0243 100.00 50.54 50.54 58.07 100.00 X X X 54.84 91.40 100.00 100.00 31.18 91.40 100.00 100.00 100.00
    T0244 100.00 50.17 100.00 95.35 39.87 38.54 100.00 56.15 100.00 98.01 100.00 90.70 100.00 98.34 100.00 100.00 100.00
    T0246 100.00 50.00 50.00 98.59 46.05 40.96 100.00 100.00 52.26 100.00 46.33 97.46 35.31 100.00 100.00 100.00 100.00
    T0247 100.00 50.00 100.00 97.80 40.11 40.93 100.00 59.07 100.00 74.73 100.00 86.26 100.00 98.35 79.95 79.95 79.95
    T0251 100.00 50.00 50.00 93.14 47.06 69.61 100.00 100.00 100.00 99.02 100.00 93.14 100.00 100.00 100.00 100.00 100.00
    T0263 100.00 50.50 100.00 97.03 54.46 73.27 100.00 100.00 100.00 62.38 100.00 66.34 100.00 96.04 100.00 100.00 100.00
    T0265 100.00 50.46 50.46 99.08 57.80 41.28 100.00 100.00 100.00 100.00 98.17 68.81 100.00 97.25 100.00 100.00 100.00
    T0266 100.00 50.00 100.00 100.00 32.24 67.11 100.00 100.00 100.00 98.68 100.00 85.53 65.79 100.00 100.00 100.00 100.00
    T0267 100.00 50.00 50.00 100.00 32.24 67.11 100.00 100.00 100.00 98.68 100.00 48.57 65.79 100.00 100.00 100.00 100.00
    T0271 100.00 50.31 100.00 99.38 29.81 63.35 100.00 100.00 100.00 78.26 100.00 96.89 100.00 100.00 100.00 100.00 100.00
    T0274 100.00 50.31 50.31 100.00 32.08 66.67 100.00 100.00 100.00 100.00 54.09 94.34 100.00 100.00 100.00 100.00 100.00
    T0275 100.00 50.37 100.00 99.27 38.69 73.72 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00
    T0276 100.00 50.00 50.00 94.02 48.91 42.39 100.00 100.00 100.00 78.26 44.02 89.13 26.09 100.00 100.00 100.00 100.00
    T0277 100.00 50.42 100.00 91.60 48.74 53.78 100.00 100.00 100.00 98.32 100.00 97.48 45.38 100.00 100.00 100.00 100.00
    T0281 100.00 50.00 50.00 85.71 100.00 58.57 100.00 100.00 100.00 100.00 X 78.57 100.00 65.71 100.00 100.00 100.00
    T0282 100.00 50.00 100.00 97.59 X 63.86 100.00 71.39 100.00 99.10 92.77 86.47 87.95 95.48 100.00 100.00 100.00
    Average 100.00 50.16 75.68 91.66 47.19 59.53 84.61 80.06 95.00 88.96 79.68 81.64 76.15 93.99 93.23 93.76 95.30
    *X: No prediction

  7. Plot of overlap score vs the percentage of correctly predicted single-domain targets
  8. Overlap Score (Two-Domain Targets)
  9. Targets Control1 Control2 Random Adda Armadillo Biozon Dompred-domssea Dompred-dps Dompro Dopro Globplot Interproscan Mateo Ssep-domain Robetta-ginzu Robetta-rosettadom Consensus
    T0199 74.56 75.15 74.56 74.56 X 98.82 91.42 92.01 74.56 69.53 74.56 85.50 70.41 88.17 92.01 63.61 89.94
    T0202 61.85 87.95 87.95 61.85 X 61.04 61.85 61.85 61.85 61.85.62 61.04 56.63 54.62 61.85 43.37 67.07 61.85
    T0209 50.63 99.58 50.63 87.87 60.25 75.73 50.63 87.87 85.77 79.50 49.79 43.52 67.78 88.70 89.12 50.63 89.12
    T0216 51.49 98.39 98.39 51.49 66.21 54.02 51.49 79.77 75.63 97.47 72.18 44.37 55.63 73.33 63.91 86.44 99.08
    T0222 84.72 65.42 84.72 84.72 59.25 34.32 98.39 87.94 84.72 84.72 84.72 80.97 84.72 96.78 93.30 93.30 93.30
    T0223 58.25 91.75 91.26 58.25 84.47 89.81 58.25 58.25 58.25 58.25 57.77 58.25 58.25 58.25 58.25 58.25 58.25
    T0228 65.04 84.85 65.04 64.80 50.58 54.55 95.80 46.85 55.94 82.75 64.57 62.01 63.40 61.77 85.32 85.32 77.62
    T0229 68.12 81.88 81.88 68.12 92.03 68.12 68.12 68.12 68.12 68.12 66.67 65.94 84.06 68.12 94.93 94.93 68.12
    T0232 63.98 86.02 63.98 68.22 85.17 65.25 97.46 63.98 63.98 77.97 63.14 49.58 63.98 84.32 99.58 99.58 95.76
    T0233 83.43 66.58 66.58 82.87 20.99 59.67 89.78 96.69 97.51 87.02 82.87 83.15 74.86 90.06 95.30 95.30 94.48
    T0235 75.95 73.95 75.95 64.13 59.92 65.33 75.95 75.75 53.51 91.18 75.95 87.98 48.70 96.59 47.90 47.90 47.90
    T0249 61.72 88.04 88.04 61.72 63.64 68.42 93.30 61.72 61.72 73.68 61.72 43.54 75.60 91.39 97.61 96.65 99.52
    T0262 67.97 82.03 67.97 67.97 67.58 68.36 67.97 67.97 89.45 92.58 56.64 63.28 67.97 60.16 68.36 70.31 83.98
    T0264 60.54 89.46 89.46 60.54 55.10 67.69 60.54 70.75 60.54 47.28 60.54 40.82 54.42 60.54 98.30 98.30 79.93
    T0269 58.80 91.20 58.80 78.80 44.00 72.40 58.80 96.80 58.80 70.00 57.60 56.00 80.00 82.00 92.00 92.00 91.60
    T0272 59.24 90.52 90.52 59.24 82.46 98.58 59.24 59.24 59.24 59.24 58.29 59.24 41.23 59.24 59.24 93.37 59.24
    T0273 79.14 70.59 79.14 79.14 56.15 62.57 79.14 79.14 79.14 79.14 55.62 79.14 79.14 79.14 79.14 75.94 79.14
    Average 66.20 83.73 77.35 69.08 55.75 68.51 74.01 73.81 69.93 75.31 64.92 62.35 66.16 76.49 79.86 80.52 80.52
    *X: No prediction

  10. Plot of overlap score vs the percentage of correctly predicted two-domain targets
  11. Plot of average overlap score vs the percentage of correctly predicted total targets

  12. Prediction Performance Separately on HM and FR targets
  13. Number of HM targets: 27
    Number of HM targets which are single-domain: 20
    Number of HM targets which are two-domain: 7

    Number of FR targets: 31
    Number of FR targets which are single-domain: 21
    Number of FR targets which are two-domain: 10

MethodsHM TargetsFR Targets
Number of targets predicted correctly as single-domainNumber of targets predicted correctly as two-domainNumber of targets predicted correctly as single-domainNumber of targets predicted correctly as two-domain
Control1200210
Control207010
Random12494
ADDA171182
Armadillo1232
Biozon1234
Dompred-Domssea182153
Dompred-DPS144144
Dompro172184
Dopro165194
Globplot181162
Interproscan193191
Mateo111101
Ssep-domain194194
Robetta-Ginzu174165
Robetta-Rosettadom174178
CONSENSUS184197

Plot of Sensitivity vs Specificity for HM Targets

Plot of Sensitivity vs Specificity for FR Targets