Skip to content
This repository was archived by the owner on Sep 17, 2023. It is now read-only.

Cloudant query support#54

Merged
danielwertheim merged 1 commit intodanielwertheim:developfrom
indranilatcal:develop
Dec 20, 2014
Merged

Cloudant query support#54
danielwertheim merged 1 commit intodanielwertheim:developfrom
indranilatcal:develop

Conversation

@indranilatcal
Copy link
Contributor

Added code to create a simple index with integration tests using Cloudant query API #53. Before adding more features, wanted to have a quick review of the components. We don't have querying yet. If okay, will add delete/listing of indices, followed by query.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not saying that this is the case, just sharing.... If you feel that there's a need for verb specific requests, e.g. PutIndexRequest that is fine. Rather that than a to generic one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about PutIndexRequest but later on removed the prefix thinking that I might be able to reuse it for delete as well (for that too, it takes .similar params in request and returns just a success/failure msg).

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But does a DELETE need all that fields? MyCouch tries to stay away from generic requests so that it's clear what "params" exists for each request. I would start specific.

@danielwertheim
Copy link
Owner

Looks nice! Great work! Added some inline feedback. BTW, if there are something you think should be changed in feature releases to make it simpler to work with, feel free to ping me at: myfirstname@lastname.se

@indranilatcal
Copy link
Contributor Author

Added code & tests for deleting indexes, listing of indexes up next. #53

@indranilatcal
Copy link
Contributor Author

Added listing of indexes. #53

@indranilatcal
Copy link
Contributor Author

Fields for an index follows sort syntax. Same does fields specified for sort during a find. I created a serializable class named Indexfield. Wondering if that should be named as SortableField instead as it can be used both to specify index fields while creating them or sorting fields during a find. Thoughts?
Another thing, can we add testenvironments.json to .gitignore and remove it from repository so that we don't have to edit it locally and remember to check it in accidentally?

@danielwertheim
Copy link
Owner

If both share the same contract, yes, if not then have two different classes.

Yes, the samples environments is probably the only thing that should be in.

@indranilatcal
Copy link
Contributor Author

Added nascent support for implicit query operators. Need to figure out a way of specifying $ operators with C# objects. #53

@danielwertheim
Copy link
Owner

Or just make it string based as SearchExpression is for searches. Then we
could add "fluff" later on but people could still use it now.
On Sep 11, 2014 10:41 PM, "indranilatcal" notifications@github.com wrote:

Added nascent support for implicit query operators. Need to figure out a
way of specifying $ operators with C# objects. #53
https://github.com/danielwertheim/mycouch/issues/53


Reply to this email directly or view it on GitHub
#54 (comment).

@indranilatcal
Copy link
Contributor Author

Till I find a better way, this is it. Which is to find by raw json. Not satisfied with it as it shifts the onus of building a valid Json back to the consumer. #53

@indranilatcal
Copy link
Contributor Author

Added some code to illustrate use of a selector object to build selector expression. Code all clubbed into a single file. If deemed fit, can organize properly and augment for other operators. #53 . Feedback awaited.

@danielwertheim
Copy link
Owner

Sorry for going MIA. Maybe some fluent config or something? I'm thinking (in the future) of pulling out the LambdaExpression parser I had in SisoDB and allow e.g. i => i.Age.Gt(32)

@indranilatcal
Copy link
Contributor Author

I added code to illustrate a config approach using enums to represent
operators. If it looks feasible. let me know.

On Wed, Sep 17, 2014 at 12:06 PM, Daniel Wertheim notifications@github.com
wrote:

Sorry for going MIA. Maybe some fluent config or something? I'm thinking
(in the future) of pulling out the LambdaExpression parser I had in SisoDB
and allow e.g. i => i.Age.Gt(32)


Reply to this email directly or view it on GitHub
#54 (comment).

@indranilatcal
Copy link
Contributor Author

I had an unrelated question for you. Is it possible to index a document
field which is an array? So that one could query something like
doc.arrayfield.contains(somevalue)

On Wed, Sep 17, 2014 at 12:06 PM, Daniel Wertheim notifications@github.com
wrote:

Sorry for going MIA. Maybe some fluent config or something? I'm thinking
(in the future) of pulling out the LambdaExpression parser I had in SisoDB
and allow e.g. i => i.Age.Gt(32)


Reply to this email directly or view it on GitHub
#54 (comment).

@danielwertheim
Copy link
Owner

In your map, just loop your nested array and emit "keys" from it.

@indranilatcal
Copy link
Contributor Author

Okay, so it needs to be a map reduce view (without reduce) rather than an
index then.

On Wed, Sep 17, 2014 at 12:28 PM, Daniel Wertheim notifications@github.com
wrote:

In your map, just loop your nested array and emit "keys" from it.


Reply to this email directly or view it on GitHub
#54 (comment).

@danielwertheim
Copy link
Owner

Not sure if the query API has support for it.
On Sep 17, 2014 10:14 AM, "indranilatcal" notifications@github.com wrote:

Okay, so it needs to be a map reduce view (without reduce) rather than an
index then.

On Wed, Sep 17, 2014 at 12:28 PM, Daniel Wertheim <
notifications@github.com>
wrote:

In your map, just loop your nested array and emit "keys" from it.


Reply to this email directly or view it on GitHub
#54 (comment).


Reply to this email directly or view it on GitHub
#54 (comment).

@indranilatcal
Copy link
Contributor Author

Thx for the idea. Applied a similar approach to the index function.
Something like for(var I=0;i<doc.items.length;i++) index('item',
doc.items[I]); Seems to work.

On Wed, Sep 17, 2014 at 2:06 PM, Daniel Wertheim notifications@github.com
wrote:

Not sure if the query API has support for it.
On Sep 17, 2014 10:14 AM, "indranilatcal" notifications@github.com
wrote:

Okay, so it needs to be a map reduce view (without reduce) rather than
an
index then.

On Wed, Sep 17, 2014 at 12:28 PM, Daniel Wertheim <
notifications@github.com>
wrote:

In your map, just loop your nested array and emit "keys" from it.


Reply to this email directly or view it on GitHub
<
https://github.com/danielwertheim/mycouch/pull/54#issuecomment-55856025>.


Reply to this email directly or view it on GitHub
#54 (comment).


Reply to this email directly or view it on GitHub
#54 (comment).

@danielwertheim
Copy link
Owner

Should I merge this in or is there work left?

@indranilatcal
Copy link
Contributor Author

Unfortunately I've not been able to spend time on this since then. I think
for now, I can leave out the structured query part and just have it with
string expressions to have it at a consistent point. If time permits, can
look at enhancing it further. Let me know if that works.

On Mon, Sep 22, 2014 at 12:22 PM, Daniel Wertheim notifications@github.com
wrote:

Should I merge this in or is there work left?


Reply to this email directly or view it on GitHub
#54 (comment).

@indranilatcal
Copy link
Contributor Author

Hi Daniel

It's been a while that I've not been able to contribute to this due to
other engagements. I reach out to you once again with a question whether
"lists" are supported in the package mycouch.cloudant. If yes, which
interfaces should I look out for?
I have a need to use it from a .NET app so any pointers are more than
welcome.

Also, somehow, I don't quite find documentation of this endpoint in
cloudant documentation either, in case you find any link from cloudant or
couchdb docs on them, please share.

Regards
Indranil

On Tue, Sep 23, 2014 at 12:52 PM, Indranil Chatterjee <
indranilatcal@gmail.com> wrote:

Unfortunately I've not been able to spend time on this since then. I think
for now, I can leave out the structured query part and just have it with
string expressions to have it at a consistent point. If time permits, can
look at enhancing it further. Let me know if that works.

On Mon, Sep 22, 2014 at 12:22 PM, Daniel Wertheim <
notifications@github.com> wrote:

Should I merge this in or is there work left?


Reply to this email directly or view it on GitHub
#54 (comment)
.

@indranilatcal
Copy link
Contributor Author

Sorry, I didn't realize that the query code wasn't merged to develop yet. I've merged it to my develop fork.
Quick summary of query support:
It supports querying by using string expressions.
Support added for creating, deleting & listing indexes.
Let me know if this pull request can be merged at its current state and if we can address fluent API for queries later by a separate pull request.

@danielwertheim
Copy link
Owner

So this PR contains both queries and lists?

I'll get to it during this weekend.

@indranilatcal
Copy link
Contributor Author

This one does not contain lists. It contains queries (with string
expressions) and index support.
List code lies in a separate branch. I might have done a comparison with a
wrong branch while creating the pull request for lists. I'd fix that and
create a new pull request for lists.

On Fri, Dec 12, 2014 at 1:57 AM, Daniel Wertheim notifications@github.com
wrote:

So this PR contains both queries and lists?

I'll get to it during this weekend.


Reply to this email directly or view it on GitHub
#54 (comment).

@indranilatcal
Copy link
Contributor Author

Hi Daniel

I've merged code for lists along with query for you to have a look at them
in tandem.
Have and look once you get time and give your feedback.

Indranil

On Fri, Dec 12, 2014 at 2:03 AM, Indranil Chatterjee <
indranilatcal@gmail.com> wrote:

This one does not contain lists. It contains queries (with string
expressions) and index support.
List code lies in a separate branch. I might have done a comparison with a
wrong branch while creating the pull request for lists. I'd fix that and
create a new pull request for lists.

On Fri, Dec 12, 2014 at 1:57 AM, Daniel Wertheim <notifications@github.com

wrote:

So this PR contains both queries and lists?

I'll get to it during this weekend.


Reply to this email directly or view it on GitHub
#54 (comment)
.

@indranilatcal
Copy link
Contributor Author

Let me know if you were able to view the changes. I included list changes
with the existing query pull request.

On Fri, Dec 12, 2014 at 11:17 AM, Indranil Chatterjee <
indranilatcal@gmail.com> wrote:

Hi Daniel

I've merged code for lists along with query for you to have a look at them
in tandem.
Have and look once you get time and give your feedback.

Indranil

On Fri, Dec 12, 2014 at 2:03 AM, Indranil Chatterjee <
indranilatcal@gmail.com> wrote:

This one does not contain lists. It contains queries (with string
expressions) and index support.
List code lies in a separate branch. I might have done a comparison with
a wrong branch while creating the pull request for lists. I'd fix that and
create a new pull request for lists.

On Fri, Dec 12, 2014 at 1:57 AM, Daniel Wertheim <
notifications@github.com> wrote:

So this PR contains both queries and lists?

I'll get to it during this weekend.


Reply to this email directly or view it on GitHub
#54 (comment)
.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just verifying, ddoc is this correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. ddoc is the property name (
https://docs.cloudant.com/api/cloudant-query.html#list-all-indexes - this
lists the response body properties).

On Fri, Dec 19, 2014 at 2:54 AM, Daniel Wertheim notifications@github.com
wrote:

In source/projects/MyCouch.Cloudant.Net45/Responses/CloudantJsonScheme.cs
#54 (diff)
:

@@ -0,0 +1,13 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+
+namespace MyCouch.Cloudant.Responses
+{

  • public static class CloudantJsonScheme
  • {
  •    public const string DesignDoc = "ddoc";
    

Just verifying, ddoc is this correct?


Reply to this email directly or view it on GitHub
https://github.com/danielwertheim/mycouch/pull/54/files#r22072432.

@danielwertheim
Copy link
Owner

Comparing stuff and just making some minor changes. Looks nice. Haven't executed the tests yet. Are there test-cases for this?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall this only take one serializer? Should it not also take the "Document configured serialzer" for use in e.g. FindResponseFactory? Look at Searches and the SearchIndexResponseFactory.

You don't need to change. Just let me know.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I need some clarity around where to use this over or in addition
to the other one.
Both of them get passed as ISerlaizer instances. Are they required for APIs
where we support strongly types entities in return (i.e. where we can use
MaterializeSuccessfulResponse with TIncludedDoc as something
other than string?
I was able to return a strongly typed entity in a test (the test
Query_can_skip_and_limit_results in the file QueryTests.cs does it.

I saw this getting used in generating requests where we serialize Json docs
into querystring parameters. Should we use it for find API (there are cases
where we use into Serializer.ToJson or ToJsonArray)?

On Fri, Dec 19, 2014 at 3:02 AM, Daniel Wertheim notifications@github.com
wrote:

In source/projects/MyCouch.Cloudant.Net45/Contexts/Queries.cs
#54 (diff)
:

+using MyCouch.Serialization;
+using System.Net.Http;
+using System.Threading.Tasks;
+
+namespace MyCouch.Cloudant.Contexts
+{

  • public class Queries : ApiContextBase, IQueries
  • {
  •    protected PostIndexHttpRequestFactory PostIndexHttpRequestFactory { get; set; }
    
  •    protected GetAllIndexesHttpRequestFactory GetAllIndexesHttpRequestFactory { get; set; }
    
  •    protected DeleteIndexHttpRequestFactory DeleteIndexHttpRequestFactory { get; set; }
    
  •    protected FindHttpRequestFactory FindHttpRequestFactory { get; set; }
    
  •    protected IndexResponseFactory IndexResponseFactory { get; set; }
    
  •    protected IndexListResponseFactory IndexListResponseFactory { get; set; }
    
  •    protected FindResponseFactory FindResponseFactory { get; set; }
    
  •    public Queries(IDbClientConnection connection, ISerializer serializer)
    

Shall this only take one serializer? Should it not also take the "Document
configured serialzer" for use in e.g. FindResponseFactory? Look at
Searches and the SearchIndexResponseFactory.

You don't need to change. Just let me know.


Reply to this email directly or view it on GitHub
https://github.com/danielwertheim/mycouch/pull/54/files#r22073017.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ISerializer that is configured as "DocumentSerializer" in bootstrapper, is used for serialization with entities where Id and Rev conventions apply.

There was an IDocumentSerializer before, but the plan is to just have one, and probably instead have specific methods in that serializer, e.g. SerializeEntity vs Serialize.

@indranilatcal
Copy link
Contributor Author

They should be under
\source\tests\MyCouch.IntegrationTests.Net45\CloudantTests. The files are
IndexTests.cs (index creation/listing API), QueryTests.cs (queries).

On Fri, Dec 19, 2014 at 2:55 AM, Daniel Wertheim notifications@github.com
wrote:

Comparing stuff and just making some minor changes. Looks nice. Haven't
executed the tests yet. Are there test-cases for this?


Reply to this email directly or view it on GitHub
#54 (comment).

@danielwertheim
Copy link
Owner

Maybe I should move this to the CouchDBClient instead of Cloudant, as they have it open sourced for CouchDB https://github.com/cloudant/mango

@danielwertheim
Copy link
Owner

I've pushed a branch https://github.com/danielwertheim/mycouch/tree/feature/pr-queries-and-lists with this pull request merged. It will go in to develop. There are some TODO in the code (four of them) that needs to be clarified. If you do them, do it in that branch and make the PR against my branch. Otherwise you could just comment here on the TODOs.

@danielwertheim danielwertheim merged commit d8f71f5 into danielwertheim:develop Dec 20, 2014
@indranilatcal
Copy link
Contributor Author

I get a 404 when I click on the branch link. It also doesn't show up as a
branch on the main page. Could I be missing something?

On Sat, Dec 20, 2014 at 4:36 PM, Daniel Wertheim notifications@github.com
wrote:

Merged #54 #54.


Reply to this email directly or view it on GitHub
#54 (comment).

@danielwertheim
Copy link
Owner

I've already finished it and merged it back in develop. No worries.

I'm rebuilding Lists a bit. Making it as an option on queries against
views. E.g. c => c.WithList("my list")

Opinions?
On Dec 21, 2014 8:07 AM, "indranilatcal" notifications@github.com wrote:

I get a 404 when I click on the branch link. It also doesn't show up as a
branch on the main page. Could I be missing something?

On Sat, Dec 20, 2014 at 4:36 PM, Daniel Wertheim notifications@github.com

wrote:

Merged #54 #54.


Reply to this email directly or view it on GitHub
#54 (comment).


Reply to this email directly or view it on GitHub
#54 (comment).

@indranilatcal
Copy link
Contributor Author

Okay. Somehow, the latest changes to your develop branch doesn't get
downloaded when I do "git pull upstream develop".

Lists always operate with views so that way it makes sense. But does it
mean merging them back to Views context?
There's another angle to look at. Both lists and shows work to transform an
output (lists apply on top of view results, whereas show applies to
individual docs). When we add show to the API, do they go under the
document context or should we consider a common context that houses both
lists and views?

Answers to some of the TODO items:
Additional query parameters are arbitrary query params that can be sent to
lists, which can be inspected & used in list functions.
There's one change I made which somehow didn't reflect: In the HttpRequest
file, I initially sent json and html as possible accept headers, but for
lists they could be anything depending on what they return. So I changed
the approach to leave it at default of Json but allowed to be set
explicitly in a list request.

HttpRequest.cs:

public virtual void SetAcceptHeader(string acceptHeader)
{
Ensure.That(acceptHeader, "acceptHeader").IsNotNullOrWhiteSpace();

        Headers["Accept"] = acceptHeader;
    }

QueryListHttpRequestFactory.cs (Create method):

if(!string.IsNullOrWhiteSpace(request.ContentType))
httpRequest.SetAcceptHeader(request.ContentType);

Added a string property named ContentType in IListParameters (and
other related files like configurator) which allowsto set accept
headers explicitly.

On Sun, Dec 21, 2014 at 1:35 PM, Daniel Wertheim notifications@github.com
wrote:

I've already finished it and merged it back in develop. No worries.

I'm rebuilding Lists a bit. Making it as an option on queries against
views. E.g. c => c.WithList("my list")

Opinions?
On Dec 21, 2014 8:07 AM, "indranilatcal" notifications@github.com
wrote:

I get a 404 when I click on the branch link. It also doesn't show up as
a
branch on the main page. Could I be missing something?

On Sat, Dec 20, 2014 at 4:36 PM, Daniel Wertheim <
notifications@github.com>

wrote:

Merged #54 #54.


Reply to this email directly or view it on GitHub
#54 (comment).


Reply to this email directly or view it on GitHub
#54 (comment).


Reply to this email directly or view it on GitHub
#54 (comment).

@danielwertheim
Copy link
Owner

Will have to look into Shows before I answer that one. You can look in https://github.com/danielwertheim/mycouch/tree/feature/lists_on_view_queries how I did it with Lists. Basically. If you define a list function, you are as of now needed to use the Views.QueryRawAsync overloads. If not, you will get an NotSupportedException stating use QueryRawAsync since you make use of lists.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants