chore: transition the library to microgenerator (#62)

* chore: remove old GAPIC code for v1 API

* Regenerate the v1 API with microgenerator

* Adjust dependencies and classifiers in setup.py

* Fix types aggregation in types.py

* Adjust import paths

* Fix and adjust unit tests

* Fix and adjust system tests

* Adjust unit test coverage threshold

Not all paths are covered, not even in the generated code, thus
the adjustment is necessary.

* Fix docs build

* Adjust quickstart sample

* Adjust sample in client docstring

* Remove beta API code and docs

* Simplify synth replacement rules and regenerate

Rules conditionally matching versions other than v1 are not needed
anymore.

* Consolidate imports in google.cloud.bigquery.storage

* Use gogole.cloud.bigquery.storage as import path

* Hide async client from most import paths

* Use GAPIC client mock in ReadRowsStream tests

* Remove redundant installations in nox sessions

* Include manual classes in reference docs

* Add UPGRADING guide

* Add minor CHANGELOG improvements

Author: plamut

packages/google-cloud-bigquery-storage/.kokoro/release/common.cfg

@@ -23,14 +23,14 @@ env_vars: {
     value: "github/python-bigquery-storage/.kokoro/release.sh"
 }
 
-# Fetch PyPI password
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "google_cloud_pypi_password"
-    }
-  }
+# Fetch PyPI password	
+before_action {	
+  fetch_keystore {	
+    keystore_resource {	
+      keystore_config_id: 73713	
+      keyname: "google_cloud_pypi_password"	
+    }	
+  }	
 }
 
 # Tokens needed to report release status back to GitHub

packages/google-cloud-bigquery-storage/CONTRIBUTING.rst

@@ -80,25 +80,6 @@ We use `nox <https://nox.readthedocs.io/en/latest/>`__ to instrument our tests.
 
 .. nox: https://pypi.org/project/nox/
 
-Note on Editable Installs / Develop Mode
-========================================
-
-- As mentioned previously, using ``setuptools`` in `develop mode`_
-  or a ``pip`` `editable install`_ is not possible with this
-  library. This is because this library uses `namespace packages`_.
-  For context see `Issue #2316`_ and the relevant `PyPA issue`_.
-
-  Since ``editable`` / ``develop`` mode can't be used, packages
-  need to be installed directly. Hence your changes to the source
-  tree don't get incorporated into the **already installed**
-  package.
-
-.. _namespace packages: https://www.python.org/dev/peps/pep-0420/
-.. _Issue #2316: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2316
-.. _PyPA issue: https://github.com/pypa/packaging-problems/issues/12
-.. _develop mode: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
-.. _editable install: https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
-
 *****************************************
 I'm getting weird errors... Can you help?
 *****************************************

packages/google-cloud-bigquery-storage/README.rst

@@ -49,11 +49,14 @@ dependencies.
 
 Supported Python Versions
 ^^^^^^^^^^^^^^^^^^^^^^^^^
-Python >= 3.5
+Python >= 3.6
 
-Deprecated Python Versions
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Python == 2.7. Python 2.7 support will be removed on January 1, 2020.
+Unsupported Python Versions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Python == 2.7, Python == 3.5.
+
+The last version of this library compatible with Python 2.7 and 3.5 is
+``google-cloud-bigquery-storage==1.1.0``.
 
 
 Mac/Linux

packages/google-cloud-bigquery-storage/UPGRADING.md

@@ -0,0 +1,282 @@
+<!--
+Copyright 2020 Google LLC
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    https://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+
+# 2.0.0 Migration Guide
+
+The 2.0 release of the `google-cloud-bigquery-storage` client is a significant
+upgrade based on a [next-gen code generator](https://github.com/googleapis/gapic-generator-python),
+and includes substantial interface changes. Existing code written for earlier versions
+of this library will likely require updates to use this version. This document
+describes the changes that have been made, and what you need to do to update your usage.
+
+If you experience issues or have questions, please file an
+[issue](https://github.com/googleapis/python-bigquery-storage/issues).
+
+
+## Supported Python Versions
+
+> **WARNING**: Breaking change
+
+The 2.0.0 release requires Python 3.6+.
+
+
+## Import Path
+
+The library was moved into `google.cloud.bigquery` namespace. It is recommended
+to use this path in order to reduce the chance of future compatibility issues
+in case the library is restuctured internally.
+
+**Before:**
+```py
+from google.cloud.bigquery_storage_v1 import BigQueryReadClient
+```
+
+**After:**
+```py
+from google.cloud.bigquery.storage import BigQueryReadClient
+```
+
+
+## Enum Types
+
+> **WARNING**: Breaking change
+
+Enum types have been moved. Access them through the `types` module.
+
+**Before:**
+```py
+from google.cloud.bigquery_storage_v1 import enums
+
+data_format = enums.DataFormat.ARROW
+```
+
+data_format = BigQueryReadClient.enums.DataFormat.ARROW
+
+**After:**
+```py
+from google.cloud.bigquery.storage import types
+
+data_format = types.DataFormat.ARROW
+```
+
+Additionally, enums cannot be accessed through the client anymore. The following
+code wil _not_ work:
+```py
+data_format = BigQueryReadClient.enums.DataFormat.ARROW
+```
+
+
+## Clients for Beta APIs
+
+> **WARNING**: Breaking change
+
+Clients for beta APIs have been removed. The following import will _not_ work:
+
+```py
+from google.cloud.bigquery_storage_v1beta1 import BigQueryStorageClient
+from google.cloud.bigquery_storage_v1beta2.gapic.big_query_read_client import BigQueryReadClient
+```
+
+The beta APIs are still available on the server side, but you will need to use
+the 1.x version of the library to access them.
+
+
+## Changed Default Value of the `read_rows()` Method's `metadata` Argument
+
+The `client.read_rows()` method does not accept `None` anymore as a valid value
+for the optional `metadata` argument. If not given, an empty tuple is used, but
+if you want to explicitly pass an "empty" value, you should use an empty tuple, too.
+
+**Before:**
+```py
+client.read_rows("stream_name", metadata=None)
+```
+
+**After:**
+```py
+client.read_rows("stream_name", metadata=())
+```
+
+OR
+
+```py
+client.read_rows("stream_name")
+```
+
+
+## Method Calls
+
+> **WARNING**: Breaking change
+
+Most of the client methods that send requests to the backend expect request objects.
+We provide a script that will convert most common use cases.
+
+> One exception to this is the `BigQueryReadClient.read_rows()` which is a hand-written
+wrapper around the auto-generated `read_rows()` method.
+
+* Install the library
+
+```py
+python3 -m pip install google-cloud-bigquery-storage
+```
+
+* The script `fixup_storage_v1_keywords.py` is shipped with the library. It expects
+an input directory (with the code to convert) and an empty destination directory.
+
+```sh
+$ scripts/fixup_storage_v1_keywords.py --input-directory .samples/ --output-directory samples/
+```
+
+**Before:**
+```py
+from google.cloud import bigquery_storage_v1
+
+client = bigquery_storage_v1.BigQueryReadClient()
+
+requested_session = bigquery_storage_v1.types.ReadSession()
+requested_session.table = "projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID"
+requested_session.data_format = bigquery_storage_v1.enums.DataFormat.ARROW
+
+session = client.create_read_session(
+    "projects/parent_project",
+    requested_session,
+    max_stream_count=1,
+)
+```
+
+**After:**
+```py
+from google.cloud.bigquery import storage
+
+client = storage.BigQueryReadClient()
+
+requested_session = storage.types.ReadSession(
+    table="projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID",
+    data_format=storage.types.DataFormat.ARROW,
+)
+session = client.create_read_session(
+    request={
+        "parent": "projects/parent_project",
+        "read_session": requested_session,
+        "max_stream_count" 1,
+    },
+)
+```
+
+### More Details
+
+In `google-cloud-bigquery-storage<2.0.0`, parameters required by the API were positional
+parameters and optional parameters were keyword parameters.
+
+**Before:**
+```py
+def create_read_session(
+    self,
+    parent,
+    read_session,
+    max_stream_count=None,
+    retry=google.api_core.gapic_v1.method.DEFAULT,
+    timeout=google.api_core.gapic_v1.method.DEFAULT,
+    metadata=None,
+):
+```
+
+In the `2.0.0` release, methods that interact with the backend have a single
+positional parameter `request`. Method docstrings indicate whether a parameter is
+required or optional.
+
+Some methods have additional keyword only parameters. The available parameters depend
+on the [`google.api.method_signature` annotation](https://github.com/googleapis/python-bigquery-storage/blob/9e1bf910e6f5010f479cf4592e25c3b3eebb456d/google/cloud/bigquery_storage_v1/proto/storage.proto#L73)
+specified by the API producer.
+
+
+**After:**
+```py
+def create_read_session(
+    self,
+    request: storage.CreateReadSessionRequest = None,
+    *,
+    parent: str = None,
+    read_session: stream.ReadSession = None,
+    max_stream_count: int = None,
+    retry: retries.Retry = gapic_v1.method.DEFAULT,
+    timeout: float = None,
+    metadata: Sequence[Tuple[str, str]] = (),
+) -> stream.ReadSession:
+```
+
+> **NOTE:** The `request` parameter and flattened keyword parameters for the API are
+> mutually exclusive. Passing both will result in an error.
+
+Both of these calls are valid:
+
+```py
+session = client.create_read_session(
+    request={
+        "parent": "projects/parent_project",
+        "read_session": requested_session,
+        "max_stream_count" 1,
+    },
+)
+```
+
+```py
+response = client.create_read_session(
+    parent="projects/parent_project",
+    read_session=requested_session,
+    max_stream_count=1,
+)
+```
+
+This call is _invalid_ because it mixes `request` with a keyword argument
+`max_stream_count`. Executing this code will result in an error:
+
+```py
+session = client.create_read_session(
+    request={
+        "parent": "projects/parent_project",
+        "read_session": requested_session,
+    },
+    max_stream_count=1,
+)
+```
+
+> **NOTE:** The `request` parameter of some methods can also contain a more rich set of
+> options that are otherwise not available as explicit keyword only parameters, thus
+> these _must_ be passed through `request`.
+
+
+## Removed Utility Methods
+
+> **WARNING**: Breaking change
+
+Several utility methods such as `project_path()` and `table_path()` have been removed.
+These paths must now be constructed manually:
+
+```py
+project_path = f"project/{PROJECT_ID}"
+table_path = f"projects/{PROJECT_ID}/datasets/{DATASET_ID}/tables/{TABLE_ID}"
+```
+
+The two that remained are `read_session_path()` and `read_stream_path()`.
+
+
+## Removed `client_config` and `channel` Parameter
+
+The client cannot be constructed with `channel` or `client_config` arguments anymore,
+these deprecated parameters have been removed.
+
+If you used `client_config` to customize retry and timeout settings for a particular
+method, you now need to do it upon method invocation by passing the custom `timeout` and
+`retry` arguments, respectively.

packages/google-cloud-bigquery-storage/docs/UPGRADING.md

@@ -0,0 +1 @@
+../UPGRADING.md

packages/google-cloud-bigquery-storage/docs/conf.py

@@ -39,6 +39,7 @@
     "sphinx.ext.autosummary",
     "sphinx.ext.intersphinx",
     "sphinx.ext.coverage",
+    "sphinx.ext.doctest",
     "sphinx.ext.napoleon",
     "sphinx.ext.todo",
     "sphinx.ext.viewcode",