How to get JSON data in chunks to report on progress? [duplicate]

This question already has an answer here: Andr开发者_开发问答oid Download Progress (1 answer) Closed 2 years ago.

I need to download contact data via a REST API, which I get in JSON format. Issue is, it might be many many contacts, so I want to observe the progress (how many contacts have already been downloaded) and report back to the user (with a progress bar, the code below runs in a thread).

However, it seems that the line client.execute(getRequest); establishes the connection and downloads the whole content in one go, i.e. my InputStream reader (to fetch the content in chunks) seems to be useless. Now I'm wondering how to make it work in chunks, so I can report on the progress?

/** prepare HTML get request */
HttpGet getRequest = new HttpGet(url[0]);
getRequest.addHeader("Authorization", "OAuth " + myTokens.get_access_token());

/** execute HTML request */
DefaultHttpClient client = new DefaultHttpClient();
JSONArray records = null;
HttpResponse response = client.execute(getRequest);

/** init response handlers for input stream */
InputStream input = new BufferedInputStream(response.getEntity().getContent());

byte data[] = new byte[MAX_BUFFER_SIZE];

long totalContactsCount = -1;
int readContactsCount = 0;
int currentByteReadCount = 0;

/** read response from inpus stream */
while ((currentByteReadCount = input.read(data)) != -1) {
    String readData = new String(data, 0, currentByteReadCount);
    dataString.append(readData);

    // then +1 progress on every ...},{... (JSON object separator)
    if (readData.indexOf("},{") >= 0) {
        readContactsCount++;
    }

    // publishing the progress....
    if (totalContactsCount > 0) {
        publishProgress((int)(readContactsCount * 100 / totalContactsCount));
    }
}
input.close();

/** transform response into JSONArray */
String result = dataString.toString();

//... convert into JSONArray

---

Unfortunatelly, as it seems from the documentation on DefaultHttpClient, the execute method returns a HTTP response, already parsed, and hence, already read from the network, so you cannot plug in the input stream. That stream is prepared for you to be read, but once it has been retreived, as you have noticed. You'd have to use another strategy. For example, trying yourself the connection using the sockets API.

---

You can first ask the web service for the total number of contacts it has (say 100,000), and then ask it for individual "pages" of contacts (say 100 contacts/page). As each page arrives, you update the progress bar. If a page request fails just retry that request and continue processing. This should be a much more reliable and efficient approach than trying to get all 100k contacts in one massive request.