Python client is slow: takes 20min to query data that takes 6 seconds using a cURL command

[pjayathissa]

I am finding the python client to be very very slow

I'm querying the data using python as follows, which takes about 20min to execute

result = query_api.query_data_frame('from(bucket: "testing")'
  			'|> range(start: 2020-02-19T23:30:00Z, stop: now())'
  			'|> filter(fn: (r) => r._measurement == "awair-api")'
               , org=credentials.org)

I run the same command using a shell script which extracts the data in about 6 seconds

curl https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/query?org=org@org.com -XPOST -sS \
  -H 'Authorization: Token tokencode==' \
  -H 'Accept: application/csv' \
  -H 'Content-type: application/vnd.flux' \
  -d 'from(bucket:"testing") |> range(start: 2020-02-19T23:30:00Z, stop: now()) |> filter(fn: (r) => r._measurement == "awair-api")' >> test.csv

[bednar]

Hi @pjayathissa,

Thanks for an open issue. Could you please share an information how your data looks like (cardinality, amount,...) ?

Regards

[pjayathissa]

Attached is a zip file of the csv that was extracted when using a cURL function and appending the result to a csv
awaircopy.csv.zip

[bednar]

Thanks @pjayathissa, I am currently investigate this issue...

[bednar]

Hi @pjayathissa,

I prepared fixed version in a branch fix/pandas-performance.

If you would like to test it then install client via:

pip install git+https://github.com/influxdata/influxdb-client-python.git@fix/pandas-performance

Regards

[joranbeasley]

this is still very very slow

# this test returns 194k rows of data
query_api().query(qs,org=org) # ~28s
query_api().query_dataframe(qs,org=org) # ~31.5s
def custom_query_dataframe(qs,org):
      httpResp = query_api().query_raw(qs,org=org)
      headers = [httpResp.readline() for _ in range(3)] # stuff i dont need(I think?... not sure about "groups")
      df = pandas.read_csv(httpResp)
      return df.drop(columns=df.columns[:2]) # some extra stuff i dont need
custom_query_dataframe(qs,org=org) # ~1-2s

[bednar]

Hi @joranbeasley,

Could you share how your data looks like?

One of the possible speed up could be install a ciso8601. The ciso8601 speed up parsing dates a lot of.

pip install ciso8601

• https://github.com/influxdata/influxdb-client-python#installation

Regards

[franz101]

Painfully slow here as well with the latest dev version

[bednar]

Hi @franz101,

Could you please share an information how your data looks like - cardinality, amount, example... ?

Which version of Python do you use?

Regards

[idubinets]

I can confirm that query_api is super slow! 200k records query takes 62 seconds.

do you have some solution or workaround?

[bednar]

@idubinets, see #371 (comment)

[sarjarapu]

@bednar 's recommendation of installing the ciso8601 dependency improved my query execution time from 1.23s to 0.34s. Thank you