About Collections

Collections are one of the most important concepts in the Journyx API. Collections are lists of resources (objects) that can be queried, sorted, filtered, and paginated, and are the standard way to access objects in the Journyx system.

There is a main reference page on Collections with complete usage details. This tutorial will provide a introduction to the concepts and usage of collections in the Journyx REST API.

Resources and collections

There are two basic categories of resources in the Journyx system: individual objects, and collections (lists) of those objects.

When you need to list available objects, or search within them, you will be using resource collections. Collection resources can (and often must be) paginated, so any given request has a limit on the amount of data returned. Collections can also be filtered (searched), and sorted. Understanding pagination becomes especially important when filtering and sorting are applied, because which items appear on which "page" of results depends heavily on the sort order and any filters that were applied.

Naturally, each user's group memberships and abilities will affect which resources are visible in any given collection. So you cannot assume that any given API client has a "complete" view of any collection. (Admin level accounts are a possible exception.)

When writing client code, when possible only work with lists of objects in chunks (pages) rather than trying to contain the entire collection in memory at once. This is particularly important when dealing with things like Project collections with 100,000+ entries that may cause very poor performance on both client and server sides.

Query parameters

When querying a collection, you can use a number of query parameters to control the results you get back. These include:

• $filter - to filter the results based on certain criteria.
• $orderBy - to sort the results based on certain criteria.
• $skip - to skip a certain number of results.
• $top - to limit the number of results returned.
• $keys - to control the level of detail in the results.

Query parameters must be encoded as part of the URL when making a GET request. How to do this varies depending on the programming language or tool you are using to make the request. For example, in Python you can use the urllib.parse module to encode query parameters as shown in the example below.

Pagination

Most (but not all) REST API collections in Journyx are paginated, and there is no way to avoid dealing with client-side pagination logic. This is primarily for performance reasons, as some data sets can be very large. Pagination means that any given set of results (a collection) will be organized into pages, and only 1 page can be retrieved at a time. Unless the client requests otherwise, the server will always return the first page of the results (that is, $skip is zero.) The default page size (number of items per slice or page) can vary but is typically 1000, which is large enough to access many collections in one page.

Each set of collection results gives a $count property indicating the total number of available results, given the current filter set. It also returns two other properties: $top and $skip. The $top property tells you the number of items in the current page, and the $skip property tells you how many results were skipped over to get to the current page. (Note there is no explicit "page number" anywhere, only top and skip.) You can think of $skip as the 0-based index (offset) into the collection list for the current page, and $top is the size of the page (i.e., number of items included).

To request the next page of results, the client needs to send the same request (i.e., use the same sorting and filtering criteria, if any) except send $skip and $top request parameters that would produce the next page of results. Generally, the new $skip should be set to the previous page's $skip plus its $top. The client can also send a $top that might even be different that then server's default, but the server will enforce a maximum $top size. This can vary between resources, but is generally 1000 items per page.

However, the server makes it easy for you by giving you @nextLink property in the collection response, which is basically just the URL that you would build yourself using the logic above. If there is no @nextLink then you're on the last page of results, which can be verified by comparing $count, $top and $skip.

Likewise, there is a @prevLink property that can be used to go back to the previous page of results. This property will only be present if you are not on the first page of results.

Sorting

Another key aspect of dealing with collections is sorting the results. It is usually far more performant to have data sorting be done inside the server, and retrieved in smaller chunks, than to fetch the entire data set to the client and perform sorting there.

Collections in our REST API can be sorted by providing the $orderBy request parameter, as shown in the example below. Most collections have a default ordering (usually on pname or similar labels) that is used when you don't give an $orderBy.

Generally, you can pick any primary property of the objects in that collection to sort by. However, not every property is always sortable because they must often be implemented with special database logic. In general, the server will respond with a 400 Bad Request if it cannot support $orderBy or filter on that particular property / key.

About primary properties

The term primary property refers to fields or properties that are intrinsic to the object itself, and are not related to other objects. This is contrasted against "extended properties" which are derived from other objects.

For example, a Project has intrinsic fields like pname and modifier, but modifier_properties.fullname (the full name of the last modifier of the Project) is an extended property.

Certain extended properties can be used for sorting and filtering, but not all of them. In general, the available fields for sorting and filtering are documented in the API reference for that object type.

The format of $orderBy is as follows. Note that you will need to URL Encode strings to submit as URL parameters when making GET requests, in order to handle things like embedded spaces, quotes, commas, etc. The encoded form is not shown here.

$orderBy = pname

That would order the results by the pname field (key) in the default ordering, which is ascending (A to Z) order. You can also explicitly specify either ASC (ascending) or DESC (descending) orderings for each field:

$orderBy = pname DESC, description ASC

You can order by (sort) by multiple fields, such that two different items which match in the first field will then be sorted by any secondary fields. Simply add additional comma-separated values of orderBy specifiers.

Filtering

You can also request that a collection be filtered, so that you only get a subset of all possible resource of that type, those tha match a certain criteria that you as the client decide or need.

As with sorting, you can't necessarily filter by every single field (key) you might find in a resource, but the majority of keys should work in most APIs. More detail about sorting and filtering can be found on the Collections page.

Filtering collections is done with the $filter URL parameter, which specifies filter conditions in a little mini-language that also uses JSON encoding.

As with the $orderBy above, we are composing a potentially complex string that must be URL encoded to be submitted correctly to the server. Encoding parameters and filtering is explained in more detail on the Collections page, but here's a quick example of a non-encoded $filter parameter:

$filter = fullname contains "John" AND (is_hidden eq true OR inactive eq true)

The basic pattern a filter specifier follows is:

FieldName <operator> VALUE

The details of available filter fields and operators is documented elsewhere.

In addition, filter conditions can be grouped with parenthesis and combined with the AND and OR logical operators (which are not case-sensitive), as shown in the example above. It shows a combination of three different filter specifiers (or expressions) that are grouped with parenthesis and joined with the AND / OR operators to produce a logical query that can be applied to the collection. These parenthesis can be nested for complex expressions. A Parsing Expression Grammar is used to parse the filter expressions and translate them to SQL queries.

It is important to note that VALUE expressions must be JSON-encoded (in an exacting way), and the entire $filter expression must typically be URL encoded before sending to the server. Most importantly, all string expressions must be valid double-quoted JSON strings.

Specifying the data to return with $keys

There are other useful GET parameters associated with collections. Perhaps the most notable is $keys which can be used to control the amount of data (or level of detail) returned for each resource, which can often have a huge number of potential fields or data points. Generally, keys are divided into a few categories:

• Base keys that are intrinsic to the resource and usually appear in the response representation. For instance, the name of a project, or the ID of the project owner.
• Extended keys that contain detail from another related resource. For instance, the fullname of the project owner, or other details about them.
• Custom fields, which are customer-defined fields that can be attached to many object types. These often have special semantics common to all custom fields.
• Additional details that are only provided upon request (by naming in the $keys parameter), generally because they are expensive to compute.

Collection example with filtering, sorting, and pagination

Here is an example that shows a complete collection request that does the following things:

• Query (GET) the /api/v1/entry_codes/codes_pay_types collection.
• Filter the results ($filter): only show items where the description key contains the text "Marketing". (Note: these string matching filters aren't case sensitive.) There are several requirements constructing filter strings discussed here, including that all string literals must be double-quoted JSON strings.
• Order the results ($orderBy): sort the results by the description field, in normal A-Z order. This is ascending ordering is implied by not saying DESC after the field name, or can be specified with ASC.
• Access the 2nd page of results ($skip=10), using 10 results per page ($top=10). Note that the page is not numbered as such, only specified by the starting point and the number of items. So, we're saying skip the first 10 items and then show me the following 10.
• Control the detail level in the resources by setting $keys to $base which means that we only want the "basic" or base keys that are inexpensive to retrieve. You can name individual fields here, and there are a few other special values that are discussed on the Collections page.

Note that VS Code is automatically taking care of the URL encoding for us here, e.g. to handle spaces and other special characters, so we don't have to worry about it here. However, in a real client program, you will likely have to take care to properly encode URLs. For example, in Python, you can encode a URL as follows:

import urllib.parse
base_url = "https://example.apps.journyx.com"
endpoint = "/api/v1/entry_codes/codes_pay_types"

params = {
  "$filter": 'description contains "Marketing"',
  "$orderBy": "description",
  "$skip": 10,
  "$top": 10,
  "$keys": "$base"
}

encoded_url = base_url + endpoint + "?" + urllib.parse.urlencode(params)
print(encoded_url)

# outputs:
# https://example.apps.journyx.com/api/v1/entry_codes/codes_pay_types?$filter=description+contains+%22Marketing%22&$orderBy=description&$skip=10&$top=10&$keys=$base

Now let's take a look at the response:

As you can see, the server reports that there are 29 total result given those filter conditions, and tells you that you're getting results 11-20, because $skip is 10, so we skipped the first 10 results and started here on result number 11, and we're including ten results here because $top is 10. So in effect, this is page two (assuming the 10 items per page we asked it for), and the server provides a handy link to page three via the @nextLink. Notice that the @nextLink includes all the filter and sorting parameters we originally supplied, because those would be important to reproducing a consistent slice of the collection that we're paginating through. This is part of why REST is "stateless" - the server doesn't know or care at any given time which "page" we're on. We have to tell it each time. Then, of course, there is the possibility of additional items being added to the collection in between your paged requests, which could really complicate things!

The results key of the response includes the actual resources (or rather, their representations) that are part of this collection, 10 of them in total as discussed above. Some of them are shrunken down in the screenshot above, so you only see the complete value for one of them. This represents the base collection of keys (fields) of this resource type, which is the "Activity" (or codes_tasks) entry code type. These objects also have additional data associated with them that might be useful, but in this example we requested only the $base keys in order to get the results faster, because we don't need that additional data right now. This helps improve performance when making a lot of calls.

Important fields in objects

In the screenshot above, you can see the complete representation of an "Activity" entry column value named "Reporting". (Actually, in this example it's an "Activity1" due to customized label testing.) It has a number of fields (keys), but we're only seeing the basic ones here due to saying $keys=$base in the request.

However, all resources (objects) typically have a few common keys:

• id - the internal ID of the resource. Usually the actual SQL table column will be named something else like id_code in this case, and that will be included as a well under another key, but for consistencies's sake, we always include an id field no matter what it's name, which can simplify a lot of client programming.

• uri - the (server-relative) URI (a.k.a. URL) of this particular resource, which is shown here as part of a collection. This tells us the address of this specific resource so we can address it with other HTTP methods like GET, PUT to modify it, or DELETE. Using GET with this URL would essentially return just this one resource, instead of a slice or page from a collection.

• tablename - we're also told which SQL table this resource belongs to, which is often useful information.

• table_label - the standard publicly-visible label for this resource type, in this case "Activity1" (for some reason).

• pname - this is commonly, but not always the main label or name of the resource, though occasionally resources either don't have such labels, or they can be found in other fields like fullname for users. This is more a quirk of the Journyx database design history more than anything.

In this case, there's another useful field shown here that won't be present in most other object types: entry_columns_uri. This provides the address (URL) of the resource for the metadata about the "Activity1" column itself. This is an example of the importance of providing links to other related resources.

Up next

Next we will be discussion creating, updating, and deleting resources in the Journyx API.