The effects of incomplete protein interaction data on structural and evolutionary inferences
© de Silva et al. 2006
Received: 01 June 2006
Accepted: 03 November 2006
Published: 03 November 2006
Skip to main content
© de Silva et al. 2006
Received: 01 June 2006
Accepted: 03 November 2006
Published: 03 November 2006
Present protein interaction network data sets include only interactions among subsets of the proteins in an organism. Previously this has been ignored, but in principle any global network analysis that only looks at partial data may be biased. Here we demonstrate the need to consider network sampling properties explicitly and from the outset in any analysis.
Here we study how properties of the yeast protein interaction network are affected by random and non-random sampling schemes using a range of different network statistics. Effects are shown to be independent of the inherent noise in protein interaction data. The effects of the incomplete nature of network data become very noticeable, especially for so-called network motifs. We also consider the effect of incomplete network data on functional and evolutionary inferences.
Crucially, when only small, partial network data sets are considered, bias is virtually inevitable. Given the scope of effects considered here, previous analyses may have to be carefully reassessed: ignoring the fact that present network data are incomplete will severely affect our ability to understand biological systems.
Molecular networks such as protein interaction, transcriptional or metabolic networks are widely seen as integrative and coherent descriptions for the whole complement of molecular processes inside a cell . There has been considerable interest in their structure, their functional organization and their evolutionary properties. For important model organisms such as Saccharomyces cerevisiae, Caenorhabditis elegans and Drosophila melanogaster there are now extensive protein interaction data deposited in public-domain databases and serious attempts are being made at elucidating the human protein interaction network (PIN) [2, 3]. These network data sets – extensive though they are thanks to experimental advances and in silico prediction – do not cover the entire network. In particular they do not include all the proteins in these organisms and represent samples only from much larger networks.
But a network introduces a set of relationships and potential dependencies between the constituent nodes and these may be broken up in the subnet. By subnet we mean a subset of the nodes of the overall global network and the interactions among them (i.e. the induced subgraph of a set of nodes); depending on how the nodes in are chosen, properties of will be different from those of . Until very recently, all studies surprisingly ignored the effects of the incompleteness of molecular networks  despite the fact that the sampling properties of networks can lead to systematic differences between the properties of networks and their subnets (discrepancies can be further inflated when the nodes in are chosen in a highly ascertained manner ). While random subnets of classical random graphs have properties that can be taken as representative of the true network, most networks, notably the popular scale-free classes of networks, will display noticeable and qualitative differences between networks and their subnets. This early work was followed by an analysis of Han et al. , who reported results regarding the effects of sampling on the degree distribution of PINs and further theoretical studies by Lee et al. ; Hakes et al.  considered not subsampling but the question of the effect of data-set selection on structural inferences of networks, which can also have considerable impact on the analysis and may explain differences between analyses. A host of other network statistics can be considered in addition to the degree distribution, Pr(k), in order to assess the structure ; these include the clustering coefficient and network motifs (see Methods for definitions). Importantly, all of these will be different for subnets compared with the true network and it is essential to understand the extent to which subnet properties other than the degree distribution differ from those of the true network. As we will show, this is to a large extent a question of how the subnet is created (that is, how nodes are chosen), and the statistic under consideration. A useful general premise we have found is that subnetworks differ more from the true network in non-local properties: i.e. their degree distributions will be more "similar" (in a loose sense which has been made somewhat more precise [5, 10]) than, for example, motif spectra [11, 12].
It is thus important to understand the extent to which the sampling properties of networks affect our inferences regarding structure, function and evolution. Considerable effort has been invested in understanding, for example, the functional organization and evolutionary properties of PINs, and contradictory results have been reported in the literature which are probably affected by many factors in addition to incomplete data. We have recently studied statistical sampling properties of network ensembles [4, 5] in considerable detail: the results suggest that when ≳80% of the nodes in a network are sampled at random, the shape of the degree distribution of the subnet, Pr*(k), will be virtually indistinguishable from that of the true network. Current PIN data comprise interactions only among a relatively small number of the proteins known to be present in the different organisms. For S. cerevisiae, for which sampling is most complete, present publicly available data sets include interaction data among ≈4900 out of an estimated 6000 proteins. We have therefore taken the present S. cerevisiae PIN as a starting point for our analysis. We compare results for subnets with those of the assumed 'true' network. This study is meant as a qualitative investigation into how incomplete sampling has affected studies into PINs and not as a quantitative assessment of the reliability of the present dataset. Despite the noise in the present yeast PIN, the S. cerevisiae data will give us a more realistic representation of a true PIN than theoretical network models.
We will show that the sampling nature of a real network does indeed lead to different properties in the subnets compared with the true network. Sampling properties of networks have hitherto been largely ignored – whereas the poor data quality has attracted considerable attention [13, 14] – but may lead to large variances and biases for network statistics obtained for different subnets, and act independently of noise. In light of the present analysis it may be necessary to reevaluate previous results for biological networks. In the context of systems biology this study demonstrates both the importance of performing carefully delimited studies of well-defined aspects of systems, and the potential pitfalls of analyzing only parts or components of complex biological systems. Clearly the way the data have been collected needs to be considered before an analysis, and the sampling properties of networks need to be included in the analysis explicitly and from the outset.
Assuming random sampling of nodes leads to great simplifications in the mathematical analysis . In reality, however, experimenters are more likely to pick some proteins than others and quite generally we can assume that each protein has probability 0 <p i < 1. Then the number of nodes in the subnet is given by
Equally, we can determine the average probability of sampling a node
As N becomes large (strictly as N → ∞) it is possible to show that we can use rather than the individual p i to determine the sampling probabilities of random networks.
Uncorrelated random networks are networks which are maximally random conditional on a given degree distribution [15, 16] (thus their degree-degree correlations may be different from zero); in such a case it is possible to express expectation values of many interesting network characteristics in terms of the degree distribution Pr(k); more interestingly, the degree sequence is a sufficient (see e.g. Cox and Hinkley ) statistic for uncorrelated networks. We can straightforwardly calculate the first two moments of the degree distribution in the subnet , Pr*(k):
where <...> denotes the sample mean and p the sampling probability. These equations are true whether a network is uncorrelated or not.
As the sampling fraction increases from zero to one the sampeld network will undergo a structural phase transition [18, 19] in the limit N → ∞. One of the main consequences is the emergence of the giant connected component . This is present (for N → ∞) when the average number of next-nearest neighbours, z 2 of a random node is on average greater than the number of its nearest neighbours z 1; i.e.
z2 > z1 (5)
The number of nearest and next-nearest neighbours in a network are given by
z1 ≡ <k> (6)
Z2 ≡ <k2> - <k>, (7)
Thus the sampling fraction p for which the subnet does not have a GCC depends in an intuitive and simple manner on the properties of the overall network . For the yeast PIN considered here the GCC will cease to exist for p ≲ 0.041. For classical or Erdös-Rényi random graphs, where the degree distribution is given by a Poisson distribution with parameter λ (for large N) equation (8) means that the sampling fraction must exceed p > for a GCC to exist.
Sampling fraction and sub-network size. In the present context, the true network has been taken to be the available PIN dataset (which contains itself interactions among 4773 out of an estimated 6000 S. cerevisiae proteins). The relationship between sampling fraction p and number of edges in the subnet is quadratic . The last line shows the extrapolation from the present network to the true network size assuming random sampling.
Number of proteins
Mean number of interactions
The effects of noise on the network data are shown in Figure 2 where we have added, subtracted and rewired, respectively, a fraction of the interactions among nodes. Qualitatively, we find that the shoulder of the degree distribution (i.e. the shape of the distribution at intermediate values of the degree k) is only little affected. Particularly at low, but also at high degree, the shape of the distribution may also differ quite considerably. Thus noise should generally distort the degree distribution in a different way from the way incomplete network data do.
Figure 1B shows the spread of the average clustering coefficient in the four subnet ensembles. The horizontal line shows the empirical clustering coefficient of the full network. In the supplementary material [see Additional file 1] we show that for large uncorrelated uniform networks  the clustering coefficient does not change at all under random sampling. The systematic decrease in the average clustering coefficients with decreasing subnet size reflects the presence of degree-degree correlations (previously shown by Agrafioti et al. ) in the network data. We also also observe an increase in spread and range with decreasing subnet size. The empirical clustering coefficient (indicated by the horizontal line) is higher than the median, but falls within the distribution of clustering coefficient (C) values obtained for all subnet ensembles, suggesting that correlations in the network are not very strong. This is in contrast to Figure 1A, where the degree distribution is more globally affected. This and the behaviour of C under sampling in the giant connected component are discussed further in the supplementary material.
The dependence of betweenness or betweenness-centrality (BC; see Methods) on the sampling fraction is more subtle than that of the degree or clustering coefficient as it also depends on the global structure of the network. Thus, for example, in different subnet samples the 10 proteins with the highest BC values change much more than the 10 proteins with the highest degrees. However, a very good correlation (Kendall's τ ≳ 0.79 in the true network) between degree and BC is seen for all values of p (data not shown).
In this study we pay particular attention to the six motifs defined by four nodes in an undirected graph (illustrated at the bottom of Figure 1C). The observed range of the Z-scores (see Methods) for all motifs considered here decreases with subnet size. For each subnet size we observe considerable spread in the range of Z-scores for the different motifs shown in Figure 1C. Motifs 1,3 and 4 can have both negative and positive Z-scores depending on the sample (motif 4 has positive and negative statistically significant Z-scores even for 80% subnets in the 20 subnets studied here). For motif 6, the most highly connected, we observe the biggest spread as well as a general increase in the average Z-score with subnet size. In Figure 1D we observe that the median Z-score for motif 6 is the same in both the 20% and 40% subnets, and the 60% and 80% subnets, respectively. This is, however, entirely due to chance and to the high variance of motif Z-scores in random subnets as is shown by further analyses (see supplementary material [See Additional file 1]). The importance of network data integrity and completeness is further exemplified by comparing the results in figure 1C with those in the original papers by Milo et al. [11, 12]; here effects of the choice of data set also come into play .
For the subnets constructed on the basis of protein expression data, we determined the 4-motif Z-scores. In Figure 3C it can be seen that all the motifs have similar Z-scores in the different data sets except for the fully connected 4-motif. The Z-scores of this motif do not exhibit a simple ordering, e.g. the subnet comprising the 80% of nodes with the highest expression levels exhibits higher Z-scores than the subnet consisting of all nodes where expression level data is available. Finally, this network has a Z-score for motif 6 that is twice as high as that obtained for the full network. We also detect some systematic differences for motifs 1, 3 and 4. These had Z-scores ≈ 0 in the true network and all randomly generated subnets (Figure 3C), but have negative Z-scores in the networks which are based on expression level. This suggests that experimental bias in designing interactome mapping studies will lead to systematic differences in motif spectra for different sampling schemes.
So far, we have considered only structural properties of networks. The interest in molecular networks lies, however, in the hope that they can explain the mechanisms underlying complex biological processes. Their impact on the evolutionary properties of molecules has also been studied and here we seek to understand how informative inferences from subnets are about the properties of larger networks.
Figure 4A shows the correlation and partial correlation (correcting for expression level variation) coefficients between evolutionary rate and degree for the 20%, 40%, 60% and 80% subnetworks; correlations and partial correlations are measured using Kendall's rank correlation coefficient, τ. The evolutionary rate is obtained from comparisons with six other yeast species , based on reconstructed phylogenies. There is a weak anti-correlation between evolutionary rate and degree and this anti-correlation is further weakened when expression level is taken into account in the partial correlation coefficients (blue boxplots in Figure 4). This anti-correlation strengthens somewhat in the larger subnets. There is a stronger anti-correlation (see Figure 4B) between evolutionary rate and expression level. These results suggest that the qualitative results of the work of, for example, Agrafioti et al.  – at least those referring to single nodes – remain valid in the ensembles of random subnets. Quite generally, under random sampling of nodes, single-node properties or any qualities that depend on a protein's degree should also be observable in the subnet. For example, under random sampling the most common proteins will remain the same, provided, of course, that they are included in the subnet (Table 2). Because of random sampling, a node which has rank m in the list of nodes ordered by degree in the full network will have rank l < m in a subnet with probability
Proteins with maximal degree-rank. The rank of a protein in the list of proteins ordered by degree, gene name, and number of connections of the top-ten most connected proteins in the full network are listed, followed by their corresponding mean rankings from the ensemble of 1000 20%, 40%, 60% and 80% subnetworks. Value in brackets are the number of subnets (out of 1000) in which the protein was present. The final column shows the degree in the high-confidence DIP data set; the correspondence between the degrees of a protein in both datasets appears to be poor. Overall, however, there is significant correlation between a protein's degree in the two data sets (τ ≈ 0.53).
Avg. rank 20% subnet
Avg. rank 40% subnet
Avg. rank 60% subnet
Avg. rank 80% subnet
Degree in high confidence data
We have explored effects of sampling on statistical measures of protein interaction structure for different sampling schemes. Our comparison with the effects of noisy interaction data (see figure 2) suggests that sampling and noise affect network statistics in different ways and we have therefore concentrated on the sampling effects as noise has received considerable attention previously (see, for example, [28, 13, 29]). Previous studies of network sampling properties focused on the degree distribution [4, 30, 6]. In our analysis we confirmed the results of these earlier studies, but one aspect of this study deserves closer scrutiny: with decreasing sampling fraction the degree distribution of the randomly sampled subnets becomes straighter and the slope of the best-fit line becomes steeper. More interestingly, we find that for a data set which had previously  been classified as consisting of more reliable interactions, the degree distribution appears to be reasonably similar to the degree distribution of the overall network (this can be also quantified statistically ), especially when compared with the randomly generated subnetwork ensemble.
Not surprisingly, we find that the effects of sampling on other statistical measures such as clustering coefficient, betweenness and motifs are more intricate (average pathlengths and diameter  have similarly diverse sampling properties). As statistical measures become less local, the effects of sampling become increasingly subtle. For example, BC is a non-local property and the effects of sampling act locally as well as globally as the system undergoes a structural phase transition with the giant connected component [19, 31] breaking up as p decrease. Thus the fraction of pairs of nodes which are connected (belong to the same component) decreases and an increasing fraction of nodes has a BC value of 0. On the other hand, the fraction of shortest paths passing through the connected nodes increases systematically.
Motifs are local objects [11, 12, 32] but Z-scores are constructed using a global network-rewiring approach [33, 34]. Therefore their sampling properties are more intricate than those of subgraphs that are defined differently . This dual nature of motifs – they are local objects but their significance is assessed against a globally randomized network ensemble – explains the qualitative differences in their behaviour under different sampling regimes.
In addition to the sampling properties, one result which becomes obvious from the present analysis is that subnets of the same size can differ quite considerably; and, in particular, the more complex measures of network structure such as motif spectra can exhibit variances that overwhelm the mean or median statistics. This becomes particularly apparent in Figure 1C. It is partially for this reason that we have not emphasised the non-random sampling schemes more: a single instance of a network statistic represents only an instance of a sample drawn from an ensemble; for networks sampling of nodes leads to very broad distributions of sample statistics as would be expected for such highly correlated and structured data sets . Sampling and noise affect these network statistics differently, with incomplete data introducing variability as well as systematic bias, and noise affecting almost exclusively the variance in, for example, the Z-scores of motifs.
For random subnets we also compared evolutionary results previously obtained for the "complete network" for the randomly generated networks. In Agrafioti et al.  only the effects of local structure (i.e. degree) were used and in light of the previous discussion it is therefore not surprising that the central results are generally confirmed in the subnets: in particular protein expression level correlates better than degree with protein evolutionary/substitution rate. For the non-random sampling schemes the data are biased in favour of protein abundance and results are also confirmed, but potentially biased somewhat against degree. In general, single-node properties of proteins are statistically conserved in the subnet, e.g. the protein with the highest degree will, provided it is being included in the sample, tend to have the highest degree also in the subnet. As far as biological and functional inferences are concerned, the effects of network sampling properties appear to be not very different from statistical missing data problems. Thus the biological studies, which investigate, for example, the interplay between protein domain structure and protein interactions  are probably not affected. Investigating such properties across a network , however, may be subject to bias because of the intricacies displayed by the network sampling behaviour discussed here.
In summary, our analysis shows that it is important to include the sampling nature of biological networks explicitly and from the outset. Failure to do so may have given rise to biases in previous network analyses. In particular this is the case for statistics which involve more than one node such as motif spectra  or pairwise similarities of nodes . In other branches of the quantitative biosciences, notably population genetics , the effects of sampling and their importance are well understood. The same is not true for the fledgling field of systems biology.
Noise and incompleteness affect network data in subtly different ways. As we have shown here, a subnet is much less than a part of the whole network and failure to account for this will bias inferences.
Protein-protein interactions of Saccharomyces cerevisiae are obtained from the DIP database which lists 4773 proteins ('nodes' in network parlance) and the 15,461 interactions observed between these proteins. It is a manually curated catalogue of protein complexes and the interactions are obtained, inter alia, from yeast two-hybrid experiments and literature extraction. It is estimated that S. cerevisiae has around 6000 genes, so that which we call the full network is really a subnetwork itself. We have removed self-interactions leaving 15,181 interacting protein pairs; self-interactions are removed so that we can describe the PIN in terms of a simple graph . It should be noted that in PINs the rates for false-positive and false-negative results are estimated [13, 39] to be around 40%, with many interactions endorsed by only one experimental observation. This dataset then constitutes our assumed "real" or complete network.
We randomly sampled (without replacement) the real network to produce 1000 subnets comprising 20%, 40%, 60% and 80% of the total number of nodes, respectively (Table 1). The random sampling scheme is the most parsimonious model for the choice of nodes which make up the subnets. In reality, however, experimentalists designing e.g. yeast two-hybrid experiments will be guided by prior knowledge and/or a particular biological question in mind. While it is difficult to model the precise ascertainment process we have some additional information which allows us to study the effects of two other ascertainment schemes: first we consider the networks generated by taking all proteins which were included in the expression analysis of Cho et al.  as well as the 20%, 40%, 60% and 80% of proteins with the highest expression levels. The second ascertainment scheme we consider is the subnet of protein interactions which have been deemed to be reliable in the analysis of Gavin et al.  (referred to in the main text as high-quality/high-confidence data).
The present S. cerevisiae PIN is, of course, not free from false-positive interactions; equally, false-negatives will have lead to missing interactions. Here we have used the PIN data as if it were the true network to study the effects of incomplete network data under different sampling schemes discussed above. In order to study the effects of noise, we follow the approach of Yook et al.  and add 10%, 20% and 40% of false interactions to study the effects of false-positives; we delete 10%, 20% and 40% of interactions to model the effect of false-negative; and we rewire (which corresponds to adding and deleting equal proportions of interactions) 10%, 20% and 40% of interactions to study the joint effects of false-positive and false-negative interactions. In this way we can qualitatively compare the effects of noise in the data with those of incomplete network data on network statistics.
The degree distribution, Pr(k), is the probability that a node has k interaction partners. In uncorrelated networks [41, 16], other properties depend only on the degree distribution and the degree sequence is a sufficient statistic. The expected degree distribution in the subnet is given by
if nodes of degree 0 in the subnet are ignored.
The clustering coefficient C is a measure of the average local neighbourhood in a network . It is defined as the probability that two nodes j and k which are connected to node i are themselves connected to each other, and its value is restricted to the unit interval, 0 ≤ C ≤ 1. It is averaged over all nodes in the network:
where k i is the degree of node i. It is a measure which describes the average local structure in a network . When C is calculated only for the giant connected component the behaviour will differ slightly (Supplementary material [See Additional file 1]).
The betweenness of a node is the number of shortest paths in a network which includes this node . Betweenness-centrality (BC) is the fraction of shortest paths which runs through a node. Here we focus on BC and its change under sampling. BC is highly correlated with degree in an obvious way with hubs having higher centrality than lower-degree nodes.
Motifs are recurring patterns of connected subgraphs. It has been speculated that motifs may represent modules that are used repeatedly in similar biological processes, just as transistors are reused in larger electronic circuits [11, 12].
Motifs and their statistical significance were determined using the mfinder package [11, 12] which randomizes the edges in the true network (in this case the S. cerevisiae full- or sub-network) among the nodes (keeping the number of nodes and the degree of each node the same as that in the true network). The frequencies of the various 4-motifs are then determined for the randomized network. This is repeated a sufficiently large number of times to give a frequency distribution for each 4-motif pattern in the ensemble of randomized networks, from which a Z-score for each motif can be determined [33, 34]; this is defined  by
here n is the number of times the motif is found in the true network is the average number of times it is found in the B replicate networks, and σ B is the standard deviation across the replicate networks. The fact that the Z-score is approximately normally distributed allows us to define p-values. Thus a Z-score of 4 already corresponds to p ≈ 3.2 × 10-5 and would suggest significant overrepresentation of the motif compared with the ensemble of randomized networks. It is therefore misleading to consider only the very highest Z-score as indicative of overrepresentation. Some authors  have argued that mere counting is sufficient to estimate the relative importance of a motif in a network. From a statistical perspective, such a notion cannot be upheld. We note, however, that the Z-score of a motif depends on an assumed probability model for network re-wiring, which may bias the Z-score.
EdeS, TT, PJI, IA and MPHS acknowledge financial support from the Wellcome Trust. CW and MPHS are grateful to the Carlsberg Foundation and the Royal Society, UK, for their generous support. CW is supported by the Danish Cancer Society. MPHS receives further support through an EMBO Young Investigator Award.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.