The study was conducted from April to August 2009 in three woredas; Omo Nada, Kersa and Tiro Afeta; in the Jimma zone of Oromia Regional State, Ethiopia (Figure 1). Within each woreda, one health centre and three health posts were purposively selected according to reported malaria cases. Jimma zone is 300 km south-west of Addis Ababa, and has an elevation of approximately 1,780 meters above sea level. Malaria transmission in Oromia is perennial, but peaks from April to May and from October to December, after the seasonal rains.

Three RDT products were compared: i) CareStart^® pf-HRP2/pan-pLDH (AccessBio, USA, catalogue number G0131SK), ii) ParaScreen^® pf-HRP2/pan-pLDH (Zephyr Biomedicals, India, catalogue number 50310025), and iii) ICT Combo^® pf-HRP2/pan-aldolase (ICT Diagnostics, South Africa, catalogue number ML02). All of the respective RDTs were from the same lot, with expiry date July 2010 (CareStart) or January 2011 (ParaScreen and ICT Combo). CareStart RDTs are individually packaged with swab, lancet, capillary tube, and buffer. ParaScreen cassettes and sample loop are individually packaged, with a single buffer bottle per box. ICT Combo cassettes are individually wrapped but buffer and all other components are provided separately in the box.

Procedures were developed in accordance with Standards for Reporting of Diagnostic Accuracy (STARD) guidelines [16]. Patients attending the outpatient department of the three health centres with symptoms of uncomplicated malaria (axillary temperature > 37.5°C or report of fever in the previous 48 hours) were recruited into the study. Patients under six months of age or with any life-threatening condition were excluded. RDTs were transported unrefrigerated by air to Addis Ababa, where they were stored at ambient conditions until transfer to field sites. Temperature was monitored (Tinytag, Gemini Data Loggers, UK) but not controlled while RDTs were transported by road to health centres and during storage at the health centres. Temperatures during transport reached a maximum 36°C, but at health facilities temperatures did not exceed 30°C.

Sample size was calculated to compare performance of each RDT to microscopy in detecting both P. falciparum and P. vivax at each site, assuming RDT sensitivity of ≥ 95% and specificity of ≥ 80%, with 80% power and a 5% significance level [17]. Assuming > 15% of febrile patients have microscopy-confirmed malaria infection, at an estimated 60:40 proportion of P. falciparum to P. vivax, a total of 811 febrile patients were estimated to be needed at each of the three sites.

Basic demographic information and clinical details were recorded from each enrolled patient. A single finger prick was performed by a laboratory technician or nurse, and used to prepare two slides, each with one thick and one thin film. The same finger prick blood sample was used to carry out all three RDTs in parallel, following manufacturer's instructions. Thin blood films were fixed in methanol after air-drying, then slides were stained in 10% Giemsa solution for 15 minutes. Thick films were read at the health centre by a laboratory technician and considered negative if no parasites were seen after examination of 200 fields at ×1,000 magnification. When positive for parasites, the number of asexual parasites per 200 white blood cells, or 500 white blood cells for low density infections, were used to calculate the number of asexual parasites per μl of blood, assuming a standard count of 8,000 white blood cells per μl of blood [18]. Species, number of parasites/μl (p/μl) and presence of gametocytes were recorded. RDT and microscopy results were read by different staff at the health centre, each blinded to the results of the other diagnostic technique.

All blood films were re-read a second time by an experienced microscopist at a regional malaria reference laboratory, blinded to initial microscopy and RDT results. A third, blinded, reading was conducted on all slides with discrepant first and second readings: presence/absence of asexual parasites, difference in species, or > 50% difference in parasite count. Microscopy results and parasite counts were corrected according to the third reading.

Each health post was staffed by two female HEWs or nurses, who received basic training on the use and interpretation of multi-species RDTs. Each carried out 50 tests with each RDT product during their normal duties both at the health post and during community outreach visits. RDTs were discarded after use. Any patient with suspected malaria was included in the sample.

Interviews were conducted with HEWs after completion of each set of 50 RDTs. Respondents were asked to grade eleven specific features of the test, such as the packaging, blood collection device, buffer and results interpretation (Additional file 2), followed by open-ended questions to probe for any further preferences. The various aspects of each RDT were graded on a scale of one (difficult) to five (very easy). After completion of all three RDT products, a final interview round was conducted to assess comparative preferences.

Heat stability assessments were conducted between August and December 2009 at the Ethiopian Health and Nutrition Research Institute located in Addis Ababa, designated as a regional reference laboratory by the World Health Organization (WHO) and the Foundation for Innovative New Diagnostics (FIND).

Heat stability and lot testing of RDTs was conducted according to the standard protocol developed by WHO/FIND [19,20]. RDTs were exposed to 35°C and 45°C for up to 90 days and to 60°C for up to 72 hours. At pre-determined periods, RDTs were removed from incubators and allowed to reach room temperature, then HRP2 and pLDH or aldolase bands were assessed individually with a negative control and using cultured P. falciparum and P. vivax at concentrations of 200 and 2,000 p/μl. For the low parasite density infection, two of each RDT were tested because of the expected lowered sensitivity, while only one of each RDT was tested at each time point with high parasite density samples.

Health centre data were entered and verified using Microsoft^® Access 2007 (Microsoft Corporation, Seattle). Second and third blood film microscopy results were entered into a Microsoft^® Excel 2007 spreadsheet (Microsoft Corporation, Seattle). All data were analyzed using STATA version 8.0 (STATA Corporation, College Station, TX, USA). RDT performance was assessed in three categories: negative, non-falciparum mono-infection, and either mixed infection with P. falciparum and a non-falciparum species, or P. falciparum mono-infection. The sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV) were calculated with 95% confidence intervals (CI) for each RDT [21]. An RDT was categorized as a true positive for P. vivax infection if only the pan band showed, while an RDT was categorized as a true negative if the test itself did not show either of the two test bands and if the corresponding blood slide was found to be negative for all Plasmodium spp. by microscopy. Similarly, an RDT was categorized as a true positive for P. falciparum or mixed infection if it showed either the HRP2 and pan band, or HRP2 band only, while it was considered as a true negative if it showed no test bands and if no Plasmodium spp. infection was diagnosed by examination of the corresponding blood slide using microscopy. The Kappa coefficient (κ), representing the proportion of agreements beyond chance, was used to quantify the level of agreement between each RDT and 'gold standard' microscopy; κ ≥ 0.8 was considered to indicate high reliability [22]. No adjustments were made for multiple comparison testing. Each of these performance indicators were compared between two RDTs using McNemar's test [23] to account for the paired nature of results.

For each of the eleven specific features of the RDTs graded by HEWs the mean score was calculated and each test received an overall rating: RDTs were ranked according to their combined score. All responses to open-ended questions given during interviews were read and coded according to common features.

The study protocol was approved by the institutional review boards of the London School of Hygiene and Tropical Medicine (Application No. 5444) and the Ethiopian Science and Technology Agency. Patients (or guardians of children ≤ 16 years) recruited at health centres provided written informed consent prior to inclusion. Study participants with microscopy-confirmed malaria infection were treated according to national guidelines [24]. At health posts, patients provided verbal consent before testing with a multi-species RDT was conducted in addition to routine ParaCheck use.

A total of 2,400 febrile patients were enrolled into the study between May 1 and July 31 2009, 800 at each of the three health centres. Data from 17 individuals were excluded due to missing RDT results or unreadable blood films, therefore full data are available from 2,383 individuals.

Five hundred and fifty two (23.2%) patients were diagnosed with malaria by microscopy, of which (297) 53.8% and (246) 44.6% were infected with P. falciparum and P. vivax respectively. Nine (1.6%) confirmed cases had mixed infection of P. falciparum and P. vivax (Figure 2). Eight individuals had gametocytes (seven P. falciparum, one P. vivax) but no asexual parasites. All 2,383 blood slides were read twice. A total of 496 slides were read a third time to address discrepancies following the first and second microscopy readings (Figure 3).

One third (38.4%) of P. falciparum cases had parasite densities > 5,000 p/μl and one-fifth (20.6%) < 200 p/μl. About half (52.6%) of patients with P. vivax had parasite density > 5,000 p/μl, while only 3.5% had low density infection (< 200 p/μl). The geometric mean (95% CI) parasite count was 1,681 p/μl (1315-2148) for P. falciparum and 3,406 p/μl (2844-4080) for P. vivax.

RDT results had moderate agreement with microscopy in detecting P. falciparum or mixed infection (κ range 0.6910 to 0.6976) and moderate to good agreement with microscopy for detection of non-falciparum malaria (κ range 0.7531 to 0.8020) (Table 1).

For some Plasmodium-infected patients the RDT result was positive but the species was incorrectly identified. The most common species misdiagnosis was where the RDT showed only the Pan band for P. falciparum mono-infections (26 CareStart, 27 ParaScreen and 25 ICT Combo). Only two of these infections were > 200 p/μl; with 12,800 p/μl and 48,000 p/μl. Additionally, RDTs occasionally indicated P. falciparum mono-infection for what was diagnosed by microscopy to be P. vivax mono-infection (three CareStart, two ParaScreen and two ICT Combo). Interestingly, all nine mixed infections resulted in HRP2 and pan band activation in CareStart and ICT Combo, while the ParaScreen pan band failed to show in one of the nine mixed infections.

The sensitivity of all RDTs in detecting P. falciparum infections was 85.6%. Specificity was statistically similar for all three RDTs at 92.4% to 92.7% (Table 1). The PPVs for the RDTs were comparable in detecting P. falciparum infection at around 65%. NPVs were the same for all RDTs at 97.5%. The false negative RDT results for P. falciparum had parasite density between 40 and 50,000 p/μl, with median density 80 p/μl. Species misdiagnoses were not considered as false negative results.

CareStart was the best performing RDT in detecting non-falciparum infections based on kappa (κ = 0.8020) (Table 1), with both CareStart and ICT Combo showing superior kappa values to that of ParaScreen (p < 0.001 and p = 0.03, respectively). The PPV and NPV of all three RDTs were comparable (p > 0.05 for all pairs), however specificity of CareStart was significantly higher than that of ParaScreen and ICT Combo (p < 0.001 and p = 0.05, respectively). False negative RDT results against P. vivax were generally low density infections: range 40 to 7080 p/μl with median 280 p/μl. Few of the false positive RDT results for P. vivax or P. falciparum were due to residual antigen from cleared infections, since only 11% of patients with false positive results had received anti-malaria treatment in the previous four weeks.

Sensitivity was calculated for P. falciparum infection and for P. vivax mono-infection at different parasite densities (< 200 p/μl of blood; 200-500 p/μl; 500-5,000 p/μl; > 5,000 p/μl) (Table 2). All three RDTs showed improved sensitivity with increasing P. falciparum density (p < 0.001), but sensitivity was generally comparable between RDTs within each parasite density category. No statistical association was seen between P. vivax density and RDT sensitivity, likely due to the small number of low density infections. However, CareStart and ICT Combo were seen to be more sensitive than ParaScreen in detection of P. vivax infection at > 5000 p/μl (p = 0.03).

All HEWs involved in the ease of use assessment had previously used ParaCheck, and commented on the importance of using a multi-species RDT in their work at the health post and in the community. "This kind of multi-species RDT is important for our routine activities and should be immediately applied in our work."

When all scored components were averaged, CareStart scored highest (4.5) followed by ParaScreen (4.4) and ICT Como (4.2). A number of HEWs indicated that they preferred the packaging of CareStart in individual 'lab in a pack' sets with all components needed for a single test (cassette, swab, lancet, blood collection pipette, buffer, and instructions) included in each packet: "I liked having all materials within a single packet, it makes moving from house-to-house easier as I don't have to carry a large box, so is very comfortable for my work in the community."

The most salient factor influencing the ease of use of each RDT was the presence of clear markings on the cassette for interpretation of results. ParaScreen had the most detailed labelling with the band positions marked with 'C' at the control region, then 'Pan' and 'Pf' in the test region. CareStart has 'C' at the control band, with '1' for P. falciparum and '2' for Pan. ICT Combo has no separate markings for Pf and Pan bands, meaning that users must remember the band location when reading results.

CareStart and ParaScreen were shown to have superior stability to that of ICT Combo during the heat stability testing. When tested after more than four hours storage at 60°C, the HRP2 band of ICT Combo failed to detect infection. The ICT Combo HRP2 band showed only very weak positive results when tested after 30 days storage at 35°C and 45°C. However, both CareStart and ParaScreen HRP2 bands were stable after storage for 72 hours at 60°C, and 90 days at 35°C and 45°C, all giving positive results when tested with P. falciparum.

All RDTs had pan bands that were more heat-sensitive than HRP2, appearing faint throughout the testing with both high and low density P. vivax infections. Both CareStart and ParaScreen pLDH bands completed the testing procedure at 35°C, 45°C and 60°C. The ICT Combo aldolase band failed to detect P. vivax at 200 p/μl after 60 days stored at 35°C and after 60 days stored at 45°C. However, after 60 days storage at these temperatures, ICT Combo did detect P. vivax at the higher density of 2,000 p/μl. ICT Combo aldolase band failed to detect 200 p/μl P. vivax after 72 hours stored at 60°C.