In the currently available EMP database (as of July 2014) [9] there are samples acquired from >200 collaborators, comprising more than 40 different biomes, defined for broad categories including marine pelagic water, freshwater lake sediment, human-associated, and so on. At a ‘30,000 feet’ perspective the EMP is identifying the environmental characteristics that correlate with microbial community structure within and between these different biomes. However, as the EMP is a collection of individual projects, each with a core hypothesis, it is also possible to discuss the immediate observations associated with individual studies. For example, exploration of human saliva from obese versus normal-weight individuals showed that while saliva was able to alter the aromatic properties of wine, only a few microbial taxa were likely to be responsible for this [10]. This preliminary study shows that oral microbes may influence the aromatic properties of food and drink, altering our satiation response. In soil systems, microbial communities from prairie soils across the Midwest of the United States of America were sequenced by the EMP. This ecosystem has been mostly replaced through agricultural land-use, and this study showed that the major shifts in their composition are driven almost exclusively by the changing relative abundance of Verrucomicrobia and its influence on carbon dynamics [11]. These analyses could be useful in helping improve prairie restoration efforts. In deep soil samples from the Russian permafrost, the EMP characterized microbial communities associated with buried organic matter, helping to identify the bacteria that were degrading the soil organic matter in these systems [12]. In deep-sea sediments from the Gulf of Mexico, the EMP data have provided understanding of how the microbial communities responded to the oil pollution from the Deepwater Horizon Oil Spill [13],[14]. Another example of investigating human impact is the analysis of freshwater river sediments along a gradient of human influence, whereby the EMP data on the microbial communities demonstrate impact-specific signals [15]. The diversity of study sites and research questions embedded in these first 30,000 samples is extraordinary, yet this is just the tip of the iceberg. Initial analysis of 10,000 of the samples identified approximately 6 million bacterial taxonomic units (genus or species level taxa), only a small fraction of which could be mapped to known phylogenies using 16S rRNA databases such as GreenGenes [16]. The frequency and distribution of these species can enable us to address interesting questions, for example, regarding the distribution of taxa across different soil ecosystems; the EMP datasets suggest that there is considerable overlap in taxa between sites, with organisms that are abundant at one location being extremely rare in another location, as previously demonstrated from marine sites [17].

A small number of concerns regarding the existing data have been raised by communities focusing on specific systems or taxa. For example, as with all studies using PCR, there are biases associated with the EMP PCR primers: they are not efficient at amplifying marine Pelagibacter ubique targets. As a result, new primers have been designed that should be more efficient in amplifying Pelagibacter, an important taxon in marine systems; however, we need to determine how efficient these new primers will be at amplifying all the other bacteria from other environments. As such, a study is underway to investigate whether rescuing Pelagibacter has deleterious consequences for other taxa or systems. However, because DNA extraction protocols themselves can have different biases depending on the environmental matrix from which the DNA is extracted [18], and PCR reagents can have contaminants that may influence amplification [19], the number of potential biases that could influence analysis is large and the key for cross-system analyses is consistent protocols. We are taking all sensible precautions to catalogue and determine potential biases: by recording all procedural and analytical variables it will be possible to determine which specific protocol elements may influence interpretation and whether the effects of these technical sources of variation limit our ability to identify important factors structuring microbial diversity.

One major challenge has been creating a master table delineating the abundance of each type of organism in each environment. With 7,000 samples for the Shenzhen meeting in 2011 [20], existing tools could barely handle the data load. In particular, the operation taxonomic unit (OTU) table, which converts the raw sequence data into a sample-by-OTU table giving the taxon abundances, strained the limits of what could be done in the traditional ‘dense’ format in which there is a slot for the abundance of each possible taxon in each environment, even if that slot has a zero count. Simply loading the table into memory and accessing specific taxa or samples became impossible as the dataset grew. Accordingly, we developed the Biological Observation Matrix (BIOM) file format [21], which reduced an early version of the EMP OTU table (6,164 samples by 7,082 OTUs) from 175 MB to 12 MB. Further improvement has been achieved by the recent move in BIOM 2.1 to HDF5, a file format used widely by physicists, climate scientists, and others needing random access to subsets of vast files. With these improvements, which are being developed fully open-source on the github repository [22], we expect that interested parties will be able to manipulate the full EMP OTU table on their laptops rather than requiring large-scale compute resources.

There are many different methods for analyzing the sequence data to obtain clusters of related sequences, each with advantages and drawbacks. For example, clustering sequences de novo produces a gold standard sequence cluster (a robust classification of a taxonomically similar group of sequences), but is very slow, while a reference-based protocol, where sequences are matched in a phylogenetic tree, is very fast but throws out sequences that fail to hit a reference. Another important challenge is visualization. QIIME [23] is the analysis architecture primarily used by the EMP, and it has long relied on KiNG [24], a molecular graphics package, for producing three-dimensional principal coordinates plots, essentially treating the community locations as atoms in a very curious molecule. However, as the size of the EMP dataset continued to grow, and the environmental contextual data became richer, the strategy of creating different views of the dataset colored by each field of contextual data (for example pH, dissolved organic carbon, and each of the hundreds of other variables captured by samples in the EMP) became unwieldy. To overcome these challenges, and to provide a three-dimensional graphics component that is directly embeddable in current web technologies, we developed EMPeror [25], software that uses current web standards such as HTML5 and OpenGL, to display even vast datasets and to explore and to recolor them dynamically.

The EMP will continue to grow and adapt as new collaborators and new technologies are added. Generating the taxon matrix in BIOM format for the existing 30,000 samples will help us to provide advice on the biomes and questions that should be targeted for the next 20,000 samples. We are also exploring metagenomic analyses for studies where the data can be used to test hypotheses regarding the ecology of microbial metabolic function (for example, [11],[13],[15]). At present, metagenomic data associated with individual studies have been made available through traditional routes (EBI, NCBI submissions), but we are working towards explicit submission and analysis pipelines for these data, including downstream analyses such as genome assemblies and metabolic pathway reconstruction. The success of the EMP has been in generating a coordinated exploration of the microbial world, and in providing the facility for data generation to collaborators who previously did not have such capacity. Primarily this has been achieved through the generation of open access data and analysis platforms that facilitate interpretation. As we move forward, we will continue to explore new avenues for collaboration, including potentially going beyond the Earth to explore extra-terrestrial locations.