python 3.x - How to manipulate nested GET calls

Question

My goal is to make multiple Get calls from the results of the first call, then concatenate the clients informations into dataframe. Preferable a faster way because I have a million clients ids

--------------------------------------
| id | name | country | city  | phone |
--------------------------------------
| 1  | Leo  | France  | Paris | 212...|
| .  |  ..  | ..      | ..    | ..    |
| 100| Bale | UK      | London| 514...|

The basic request / results (all clients):

import requests
from requests.auth import HTTPBasicAuth

# the initial request which returns all clients
res0 = requests.get('https://x.y.z/api/v1/data/clients', auth=HTTPBasicAuth('me', 'blabla'))

# results
<?xml version="1.0" ?>
<queryResponse>
  <entityId type="clients" url="https://x.y.z/api/v1/data/clients/1">1</entityId>
  ...
  ...
  <entityId type="clients" url="https://x.y.z/api/v1/data/clients/100">100</entityId>
</queryResponse>

The detailed request / results (client infos)

# this request allows to get client informations
res1 = requests.get('https://x.y.z/api/v1/data/clients/1', auth=HTTPBasicAuth('me', 'blabla'))

# results
<queryResponse>
<entity type="client_infos" url="https://x.y.z/api/v1/data/clients/1">
    <client_infos displayName="1" id="1">
        <name>Leo Massina</name>
        <country>France</country>
        <city>1607695021057</city>
        <phone>+212-61-88-65-123</phone>
    </client_infos >
</entity>

Answer 1

You can use lxml to parse the response and make the new calls, retrieve the tags and text in a dictionary and create the dataframe: (I used fors for clarity, you can optimize the code if needed)

Also, I did not retrieve ids, if needed they can be retrieved as attribute of client_infos tags.

from lxml import etree
root = etree.fromstring(res0)
reqentity = []
data = {"name":[], "country":[], "city":[], "phone":[]}
for entity in root.findall('./entityId'):
    reqentity.append(requests.get(entity.attrib['url'], auth=HTTPBasicAuth('me', 'blabla')))
for entity in reqentity:
    entity = etree.fromstring(entity)
    for item in entity.findall(".//client_infos//"):
        data[item.tag].append(item.text)
df = pd.DataFrame(data)