qlc(3erl) Erlang Module Definition qlc(3erl)
NAME
qlc - Query interface to Mnesia, ETS, Dets, and so on.
DESCRIPTION
This module provides a query interface to Mnesia, ETS, Dets, and other
data structures that provide an iterator style traversal of objects.
OVERVIEW
This module provides a query interface to QLC tables. Typical QLC ta-
bles are Mnesia, ETS, and Dets tables. Support is also provided for
user-defined tables, see section Implementing a QLC Table. A query is
expressed using Query List Comprehensions (QLCs). The answers to a
query are determined by data in QLC tables that fulfill the constraints
expressed by the QLCs of the query. QLCs are similar to ordinary list
comprehensions as described in Erlang Reference Manual and Program-
ming Examples, except that variables introduced in patterns cannot be
used in list expressions. In the absence of optimizations and options
such as cache and unique (see section Common Options, every QLC free of
QLC tables evaluates to the same list of answers as the identical ordi-
nary list comprehension.
While ordinary list comprehensions evaluate to lists, calling q/1,2 re-
turns a query handle. To obtain all the answers to a query, eval/1,2 is
to be called with the query handle as first argument. Query handles are
essentially functional objects (funs) created in the module calling
q/1,2. As the funs refer to the module code, be careful not to keep
query handles too long if the module code is to be replaced. Code re-
placement is described in section Compilation and Code Loading in the
Erlang Reference Manual. The list of answers can also be traversed in
chunks by use of a query cursor. Query cursors are created by calling
cursor/1,2 with a query handle as first argument. Query cursors are es-
sentially Erlang processes. One answer at a time is sent from the query
cursor process to the process that created the cursor.
SYNTAX
Syntactically QLCs have the same parts as ordinary list comprehensions:
[Expression || Qualifier1, Qualifier2, ...]
Expression (the template) is any Erlang expression. Qualifiers are ei-
ther filters or generators. Filters are Erlang expressions returning
boolean(). Generators have the form Pattern <- ListExpression, where
ListExpression is an expression evaluating to a query handle or a list.
Query handles are returned from append/1,2, keysort/2,3, q/1,2,
sort/1,2, string_to_handle/1,2,3, and table/2.
EVALUATION
A query handle is evaluated in the following order:
* Inspection of options and the collection of information about ta-
bles. As a result, qualifiers are modified during the optimization
phase.
* All list expressions are evaluated. If a cursor has been created,
evaluation takes place in the cursor process. For list expressions
that are QLCs, the list expressions of the generators of the QLCs
are evaluated as well. Be careful if list expressions have side ef-
fects, as list expressions are evaluated in unspecified order.
* The answers are found by evaluating the qualifiers from left to
right, backtracking when some filter returns false, or collecting
the template when all filters return true.
Filters that do not return boolean() but fail are handled differently
depending on their syntax: if the filter is a guard, it returns false,
otherwise the query evaluation fails. This behavior makes it possible
for the qlc module to do some optimizations without affecting the mean-
ing of a query. For example, when testing some position of a table and
one or more constants for equality, only the objects with equal values
are candidates for further evaluation. The other objects are guaranteed
to make the filter return false, but never fail. The (small) set of
candidate objects can often be found by looking up some key values of
the table or by traversing the table using a match specification. It is
necessary to place the guard filters immediately after the table gener-
ator, otherwise the candidate objects are not restricted to a small
set. The reason is that objects that could make the query evaluation
fail must not be excluded by looking up a key or running a match speci-
fication.
JOIN
The qlc module supports fast join of two query handles. Fast join is
possible if some position P1 of one query handler and some position P2
of another query handler are tested for equality. Two fast join methods
are provided:
* Lookup join traverses all objects of one query handle and finds ob-
jects of the other handle (a QLC table) such that the values at P1
and P2 match or compare equal. The qlc module does not create any
indexes but looks up values using the key position and the indexed
positions of the QLC table.
* Merge join sorts the objects of each query handle if necessary and
filters out objects where the values at P1 and P2 do not compare
equal. If many objects with the same value of P2 exist, a temporary
file is used for the equivalence classes.
The qlc module warns at compile time if a QLC combines query handles in
such a way that more than one join is possible. That is, no query plan-
ner is provided that can select a good order between possible join op-
erations. It is up to the user to order the joins by introducing query
handles.
The join is to be expressed as a guard filter. The filter must be
placed immediately after the two joined generators, possibly after
guard filters that use variables from no other generators but the two
joined generators. The qlc module inspects the operands of =:=/2, ==/2,
is_record/2, element/2, and logical operators (and/2, or/2, andalso/2,
orelse/2, xor/2) when determining which joins to consider.
COMMON OPTIONS
The following options are accepted by cursor/2, eval/2, fold/4, and
info/2:
* {cache_all, Cache}, where Cache is equal to ets or list adds a
{cache, Cache} option to every list expression of the query except
tables and lists. Defaults to {cache_all, no}. Option cache_all is
equivalent to {cache_all, ets}.
* {max_list_size, MaxListSize}, where MaxListSize is the size in
bytes of terms on the external format. If the accumulated size of
collected objects exceeds MaxListSize, the objects are written onto
a temporary file. This option is used by option {cache, list} and
by the merge join method. Defaults to 512*1024 bytes.
* {tmpdir_usage, TmpFileUsage} determines the action taken when qlc
is about to create temporary files on the directory set by option
tmpdir. If the value is not_allowed, an error tuple is returned,
otherwise temporary files are created as needed. Default is al-
lowed, which means that no further action is taken. The values
info_msg, warning_msg, and error_msg mean that the function with
the corresponding name in module error_logger is called for print-
ing some information (currently the stacktrace).
* {tmpdir, TempDirectory} sets the directory used by merge join for
temporary files and by option {cache, list}. The option also over-
rides option tmpdir of keysort/3 and sort/2. Defaults to "", which
means that the directory returned by file:get_cwd() is used.
* {unique_all, true} adds a {unique, true} option to every list ex-
pression of the query. Defaults to {unique_all, false}. Option
unique_all is equivalent to {unique_all, true}.
GETTING STARTED
As mentioned earlier, queries are expressed in the list comprehension
syntax as described in section Expressions in Erlang Reference Manual.
In the following, some familiarity with list comprehensions is assumed.
The examples in section List Comprehensions in Programming Examples
can get you started. Notice that list comprehensions do not add any
computational power to the language; anything that can be done with
list comprehensions can also be done without them. But they add syntax
for expressing simple search problems, which is compact and clear once
you get used to it.
Many list comprehension expressions can be evaluated by the qlc module.
Exceptions are expressions, such that variables introduced in patterns
(or filters) are used in some generator later in the list comprehen-
sion. As an example, consider an implementation of lists:append(L): [X
||Y <- L, X <- Y]. Y is introduced in the first generator and used in
the second. The ordinary list comprehension is normally to be preferred
when there is a choice as to which to use. One difference is that
eval/1,2 collects answers in a list that is finally reversed, while
list comprehensions collect answers on the stack that is finally un-
wound.
What the qlc module primarily adds to list comprehensions is that data
can be read from QLC tables in small chunks. A QLC table is created by
calling qlc:table/2. Usually qlc:table/2 is not called directly from
the query but through an interface function of some data structure. Er-
lang/OTP includes a few examples of such functions: mnesia:table/1,2,
ets:table/1,2, and dets:table/1,2. For a given data structure, many
functions can create QLC tables, but common for these functions is that
they return a query handle created by qlc:table/2. Using the QLC tables
provided by Erlang/OTP is usually probably sufficient, but for the more
advanced user section Implementing a QLC Table describes the implemen-
tation of a function calling qlc:table/2.
Besides qlc:table/2, other functions return query handles. They are
used more seldom than tables, but are sometimes useful. qlc:append/1,2
traverses objects from many tables or lists after each other. If, for
example, you want to traverse all answers to a query QH and then finish
off by a term {finished}, you can do that by calling qlc:append(QH,
[{finished}]). append/2 first returns all objects of QH, then {fin-
ished}. If a tuple {finished} exists among the answers to QH, it is re-
turned twice from append/2.
As another example, consider concatenating the answers to two queries
QH1 and QH2 while removing all duplicates. This is accomplished by us-
ing option unique:
qlc:q([X || X <- qlc:append(QH1, QH2)], {unique, true})
The cost is substantial: every returned answer is stored in an ETS ta-
ble. Before returning an answer, it is looked up in the ETS table to
check if it has already been returned. Without the unique option, all
answers to QH1 would be returned followed by all answers to QH2. The
unique option keeps the order between the remaining answers.
If the order of the answers is not important, there is an alternative
to the unique option, namely to sort the answers uniquely:
qlc:sort(qlc:q([X || X <- qlc:append(QH1, QH2)], {unique, true})).
This query also removes duplicates but the answers are sorted. If there
are many answers, temporary files are used. Notice that to get the
first unique answer, all answers must be found and sorted. Both alter-
natives find duplicates by comparing answers, that is, if A1 and A2 are
answers found in that order, then A2 is a removed if A1 == A2.
To return only a few answers, cursors can be used. The following code
returns no more than five answers using an ETS table for storing the
unique answers:
C = qlc:cursor(qlc:q([X || X <- qlc:append(QH1, QH2)],{unique,true})),
R = qlc:next_answers(C, 5),
ok = qlc:delete_cursor(C),
R.
QLCs are convenient for stating constraints on data from two or more
tables. The following example does a natural join on two query handles
on position 2:
qlc:q([{X1,X2,X3,Y1} ||
{X1,X2,X3} <- QH1,
{Y1,Y2} <- QH2,
X2 =:= Y2])
The qlc module evaluates this differently depending on the query han-
dles QH1 and QH2. If, for example, X2 is matched against the key of a
QLC table, the lookup join method traverses the objects of QH2 while
looking up key values in the table. However, if not X2 or Y2 is matched
against the key or an indexed position of a QLC table, the merge join
method ensures that QH1 and QH2 are both sorted on position 2 and next
do the join by traversing the objects one by one.
Option join can be used to force the qlc module to use a certain join
method. For the rest of this section it is assumed that the excessively
slow join method called "nested loop" has been chosen:
qlc:q([{X1,X2,X3,Y1} ||
{X1,X2,X3} <- QH1,
{Y1,Y2} <- QH2,
X2 =:= Y2],
{join, nested_loop})
In this case the filter is applied to every possible pair of answers to
QH1 and QH2, one at a time. If there are M answers to QH1 and N answers
to QH2, the filter is run M*N times.
If QH2 is a call to the function for gb_trees, as defined in section
Implementing a QLC Table, then gb_table:table/1, the iterator for the
gb-tree is initiated for each answer to QH1. The objects of the gb-tree
are then returned one by one. This is probably the most efficient way
of traversing the table in that case, as it takes minimal computational
power to get the following object. But if QH2 is not a table but a more
complicated QLC, it can be more efficient to use some RAM memory for
collecting the answers in a cache, particularly if there are only a few
answers. It must then be assumed that evaluating QH2 has no side ef-
fects so that the meaning of the query does not change if QH2 is evalu-
ated only once. One way of caching the answers is to evaluate QH2 first
of all and substitute the list of answers for QH2 in the query. Another
way is to use option cache. It is expressed like this:
QH2' = qlc:q([X || X <- QH2], {cache, ets})
or only
QH2' = qlc:q([X || X <- QH2], cache)
The effect of option cache is that when generator QH2' is run the first
time, every answer is stored in an ETS table. When the next answer of
QH1 is tried, answers to QH2' are copied from the ETS table, which is
very fast. As for option unique the cost is a possibly substantial
amount of RAM memory.
Option {cache, list} offers the possibility to store the answers in a
list on the process heap. This has the potential of being faster than
ETS tables, as there is no need to copy answers from the table. How-
ever, it can often result in slower evaluation because of more garbage
collections of the process heap and increased RAM memory consumption
because of larger heaps. Another drawback with cache lists is that if
the list size exceeds a limit, a temporary file is used. Reading the
answers from a file is much slower than copying them from an ETS table.
But if the available RAM memory is scarce, setting the limit to some
low value is an alternative.
Option cache_all can be set to ets or list when evaluating a query. It
adds a cache or {cache, list} option to every list expression except
QLC tables and lists on all levels of the query. This can be used for
testing if caching would improve efficiency at all. If the answer is
yes, further testing is needed to pinpoint the generators that are to
be cached.
IMPLEMENTING A QLC TABLE
As an example of how to use function table/2, the implementation of a
QLC table for the gb_trees module is given:
-module(gb_table).
-export([table/1]).
table(T) ->
TF = fun() -> qlc_next(gb_trees:next(gb_trees:iterator(T))) end,
InfoFun = fun(num_of_objects) -> gb_trees:size(T);
(keypos) -> 1;
(is_sorted_key) -> true;
(is_unique_objects) -> true;
(_) -> undefined
end,
LookupFun =
fun(1, Ks) ->
lists:flatmap(fun(K) ->
case gb_trees:lookup(K, T) of
{value, V} -> [{K,V}];
none -> []
end
end, Ks)
end,
FormatFun =
fun({all, NElements, ElementFun}) ->
ValsS = io_lib:format("gb_trees:from_orddict(~w)",
[gb_nodes(T, NElements, ElementFun)]),
io_lib:format("gb_table:table(~s)", [ValsS]);
({lookup, 1, KeyValues, _NElements, ElementFun}) ->
ValsS = io_lib:format("gb_trees:from_orddict(~w)",
[gb_nodes(T, infinity, ElementFun)]),
io_lib:format("lists:flatmap(fun(K) -> "
"case gb_trees:lookup(K, ~s) of "
"{value, V} -> [{K,V}];none -> [] end "
"end, ~w)",
[ValsS, [ElementFun(KV) || KV <- KeyValues]])
end,
qlc:table(TF, [{info_fun, InfoFun}, {format_fun, FormatFun},
{lookup_fun, LookupFun},{key_equality,'=='}]).
qlc_next({X, V, S}) ->
[{X,V} | fun() -> qlc_next(gb_trees:next(S)) end];
qlc_next(none) ->
[].
gb_nodes(T, infinity, ElementFun) ->
gb_nodes(T, -1, ElementFun);
gb_nodes(T, NElements, ElementFun) ->
gb_iter(gb_trees:iterator(T), NElements, ElementFun).
gb_iter(_I, 0, _EFun) ->
'...';
gb_iter(I0, N, EFun) ->
case gb_trees:next(I0) of
{X, V, I} ->
[EFun({X,V}) | gb_iter(I, N-1, EFun)];
none ->
[]
end.
TF is the traversal function. The qlc module requires that there is a
way of traversing all objects of the data structure. gb_trees has an
iterator function suitable for that purpose. Notice that for each ob-
ject returned, a new fun is created. As long as the list is not termi-
nated by [], it is assumed that the tail of the list is a nullary func-
tion and that calling the function returns further objects (and func-
tions).
The lookup function is optional. It is assumed that the lookup function
always finds values much faster than it would take to traverse the ta-
ble. The first argument is the position of the key. As qlc_next/1 re-
turns the objects as {Key, Value} pairs, the position is 1. Notice that
the lookup function is to return {Key, Value} pairs, as the traversal
function does.
The format function is also optional. It is called by info/1,2 to give
feedback at runtime of how the query is to be evaluated. Try to give as
good feedback as possible without showing too much details. In the ex-
ample, at most seven objects of the table are shown. The format func-
tion handles two cases: all means that all objects of the table are
traversed; {lookup, 1, KeyValues} means that the lookup function is
used for looking up key values.
Whether the whole table is traversed or only some keys looked up de-
pends on how the query is expressed. If the query has the form
qlc:q([T || P <- LE, F])
and P is a tuple, the qlc module analyzes P and F in compile time to
find positions of tuple P that are tested for equality to constants. If
such a position at runtime turns out to be the key position, the lookup
function can be used, otherwise all objects of the table must be tra-
versed. The info function InfoFun returns the key position. There can
be indexed positions as well, also returned by the info function. An
index is an extra table that makes lookup on some position fast. Mnesia
maintains indexes upon request, and introduces so called secondary
keys. The qlc module prefers to look up objects using the key before
secondary keys regardless of the number of constants to look up.
KEY EQUALITY
Erlang/OTP has two operators for testing term equality: ==/2 and =:=/2.
The difference is all about the integers that can be represented by
floats. For example, 2 == 2.0 evaluates to true while 2 =:= 2.0 evalu-
ates to false. Normally this is a minor issue, but the qlc module can-
not ignore the difference, which affects the user's choice of operators
in QLCs.
If the qlc module at compile time can determine that some constant is
free of integers, it does not matter which one of ==/2 or =:=/2 is
used:
1> E1 = ets:new(t, [set]), % uses =:=/2 for key equality
Q1 = qlc:q([K ||
{K} <- ets:table(E1),
K == 2.71 orelse K == a]),
io:format("~s~n", [qlc:info(Q1)]).
ets:match_spec_run(
lists:flatmap(fun(V) ->
ets:lookup(#Ref<0.3098908599.2283929601.256025>,
V)
end,
[a, 2.71]),
ets:match_spec_compile([{{'$1'}, [], ['$1']}]))
In the example, operator ==/2 has been handled exactly as =:=/2 would
have been handled. However, if it cannot be determined at compile time
that some constant is free of integers, and the table uses =:=/2 when
comparing keys for equality (see option key_equality), then the qlc
module does not try to look up the constant. The reason is that there
is in the general case no upper limit on the number of key values that
can compare equal to such a constant; every combination of integers and
floats must be looked up:
2> E2 = ets:new(t, [set]),
true = ets:insert(E2, [{{2,2},a},{{2,2.0},b},{{2.0,2},c}]),
F2 = fun(I) ->
qlc:q([V || {K,V} <- ets:table(E2), K == I])
end,
Q2 = F2({2,2}),
io:format("~s~n", [qlc:info(Q2)]).
ets:table(#Ref<0.3098908599.2283929601.256125>,
[{traverse,
{select,
[{{'$1', '$2'}, [{'==', '$1', {const, {2, 2}}}], ['$2']}]}}])
3> lists:sort(qlc:e(Q2)).
[a,b,c]
Looking up only {2,2} would not return b and c.
If the table uses ==/2 when comparing keys for equality, the qlc module
looks up the constant regardless of which operator is used in the QLC.
However, ==/2 is to be preferred:
4> E3 = ets:new(t, [ordered_set]), % uses ==/2 for key equality
true = ets:insert(E3, [{{2,2.0},b}]),
F3 = fun(I) ->
qlc:q([V || {K,V} <- ets:table(E3), K == I])
end,
Q3 = F3({2,2}),
io:format("~s~n", [qlc:info(Q3)]).
ets:match_spec_run(ets:lookup(#Ref<0.3098908599.2283929601.256211>,
{2, 2}),
ets:match_spec_compile([{{'$1', '$2'}, [], ['$2']}]))
5> qlc:e(Q3).
[b]
Lookup join is handled analogously to lookup of constants in a table:
if the join operator is ==/2, and the table where constants are to be
looked up uses =:=/2 when testing keys for equality, then the qlc mod-
ule does not consider lookup join for that table.
DATA TYPES
abstract_expr() = erl_parse:abstract_expr()
Parse trees for Erlang expression, see section The Abstract For-
mat in the ERTS User's Guide.
answer() = term()
answers() = [answer()]
cache() = ets | list | no
match_expression() = ets:match_spec()
Match specification, see section Match Specifications in Erlang
in the ERTS User's Guide and ms_transform(3erl).
no_files() = integer() >= 1
An integer > 1.
key_pos() = integer() >= 1 | [integer() >= 1]
max_list_size() = integer() >= 0
order() = ascending | descending | order_fun()
order_fun() = fun((term(), term()) -> boolean())
query_cursor()
A query cursor.
query_handle()
A query handle.
query_handle_or_list() = query_handle() | list()
query_list_comprehension() = term()
A literal query list comprehension.
spawn_options() = default | [proc_lib:spawn_option()]
sort_options() = [sort_option()] | sort_option()
sort_option() =
{compressed, boolean()} |
{no_files, no_files()} |
{order, order()} |
{size, integer() >= 1} |
{tmpdir, tmp_directory()} |
{unique, boolean()}
See file_sorter(3erl).
tmp_directory() = [] | file:name()
tmp_file_usage() =
allowed | not_allowed | info_msg | warning_msg | error_msg
EXPORTS
append(QHL) -> QH
Types:
QHL = [query_handle_or_list()]
QH = query_handle()
Returns a query handle. When evaluating query handle QH, all an-
swers to the first query handle in QHL are returned, followed by
all answers to the remaining query handles in QHL.
append(QH1, QH2) -> QH3
Types:
QH1 = QH2 = query_handle_or_list()
QH3 = query_handle()
Returns a query handle. When evaluating query handle QH3, all
answers to QH1 are returned, followed by all answers to QH2.
append(QH1, QH2) is equivalent to append([QH1, QH2]).
cursor(QH) -> Cursor
cursor(QH, Options) -> Cursor
Types:
QH = query_handle_or_list()
Options = [Option] | Option
Option =
{cache_all, cache()} |
cache_all |
{max_list_size, max_list_size()} |
{spawn_options, spawn_options()} |
{tmpdir_usage, tmp_file_usage()} |
{tmpdir, tmp_directory()} |
{unique_all, boolean()} |
unique_all
Cursor = query_cursor()
Creates a query cursor and makes the calling process the owner
of the cursor. The cursor is to be used as argument to next_an-
swers/1,2 and (eventually) delete_cursor/1. Calls er-
lang:spawn_opt/2 to spawn and link to a process that evaluates
the query handle. The value of option spawn_options is used as
last argument when calling spawn_opt/2. Defaults to [link].
Example:
1> QH = qlc:q([{X,Y} || X <- [a,b], Y <- [1,2]]),
QC = qlc:cursor(QH),
qlc:next_answers(QC, 1).
[{a,1}]
2> qlc:next_answers(QC, 1).
[{a,2}]
3> qlc:next_answers(QC, all_remaining).
[{b,1},{b,2}]
4> qlc:delete_cursor(QC).
ok
cursor(QH) is equivalent to cursor(QH, []).
delete_cursor(QueryCursor) -> ok
Types:
QueryCursor = query_cursor()
Deletes a query cursor. Only the owner of the cursor can delete
the cursor.
e(QH) -> Answers | Error
e(QH, Options) -> Answers | Error
eval(QH) -> Answers | Error
eval(QH, Options) -> Answers | Error
Types:
QH = query_handle_or_list()
Answers = answers()
Options = [Option] | Option
Option =
{cache_all, cache()} |
cache_all |
{max_list_size, max_list_size()} |
{tmpdir_usage, tmp_file_usage()} |
{tmpdir, tmp_directory()} |
{unique_all, boolean()} |
unique_all
Error = {error, module(), Reason}
Reason = file_sorter:reason()
Evaluates a query handle in the calling process and collects all
answers in a list.
Example:
1> QH = qlc:q([{X,Y} || X <- [a,b], Y <- [1,2]]),
qlc:eval(QH).
[{a,1},{a,2},{b,1},{b,2}]
eval(QH) is equivalent to eval(QH, []).
fold(Function, Acc0, QH) -> Acc1 | Error
fold(Function, Acc0, QH, Options) -> Acc1 | Error
Types:
QH = query_handle_or_list()
Function = fun((answer(), AccIn) -> AccOut)
Acc0 = Acc1 = AccIn = AccOut = term()
Options = [Option] | Option
Option =
{cache_all, cache()} |
cache_all |
{max_list_size, max_list_size()} |
{tmpdir_usage, tmp_file_usage()} |
{tmpdir, tmp_directory()} |
{unique_all, boolean()} |
unique_all
Error = {error, module(), Reason}
Reason = file_sorter:reason()
Calls Function on successive answers to the query handle to-
gether with an extra argument AccIn. The query handle and the
function are evaluated in the calling process. Function must re-
turn a new accumulator, which is passed to the next call. Acc0
is returned if there are no answers to the query handle.
Example:
1> QH = [1,2,3,4,5,6],
qlc:fold(fun(X, Sum) -> X + Sum end, 0, QH).
21
fold(Function, Acc0, QH) is equivalent to fold(Function, Acc0,
QH, []).
format_error(Error) -> Chars
Types:
Error = {error, module(), term()}
Chars = io_lib:chars()
Returns a descriptive string in English of an error tuple re-
turned by some of the functions of the qlc module or the parse
transform. This function is mainly used by the compiler invoking
the parse transform.
info(QH) -> Info
info(QH, Options) -> Info
Types:
QH = query_handle_or_list()
Options = [Option] | Option
Option = EvalOption | ReturnOption
EvalOption =
{cache_all, cache()} |
cache_all |
{max_list_size, max_list_size()} |
{tmpdir_usage, tmp_file_usage()} |
{tmpdir, tmp_directory()} |
{unique_all, boolean()} |
unique_all
ReturnOption =
{depth, Depth} |
{flat, boolean()} |
{format, Format} |
{n_elements, NElements}
Depth = infinity | integer() >= 0
Format = abstract_code | string
NElements = infinity | integer() >= 1
Info = abstract_expr() | string()
Returns information about a query handle. The information de-
scribes the simplifications and optimizations that are the re-
sults of preparing the query for evaluation. This function is
probably mainly useful during debugging.
The information has the form of an Erlang expression where QLCs
most likely occur. Depending on the format functions of men-
tioned QLC tables, it is not certain that the information is ab-
solutely accurate.
Options:
* The default is to return a sequence of QLCs in a block, but
if option {flat, false} is specified, one single QLC is re-
turned.
* The default is to return a string, but if option {format,
abstract_code} is specified, abstract code is returned in-
stead. In the abstract code, port identifiers, references,
and pids are represented by strings.
* The default is to return all elements in lists, but if op-
tion {n_elements, NElements} is specified, only a limited
number of elements are returned.
* The default is to show all parts of objects and match speci-
fications, but if option {depth, Depth} is specified, parts
of terms below a certain depth are replaced by '...'.
info(QH) is equivalent to info(QH, []).
Examples:
In the following example two simple QLCs are inserted only to
hold option {unique, true}:
1> QH = qlc:q([{X,Y} || X <- [x,y], Y <- [a,b]]),
io:format("~s~n", [qlc:info(QH, unique_all)]).
begin
V1 =
qlc:q([
SQV ||
SQV <- [x, y]
],
[{unique, true}]),
V2 =
qlc:q([
SQV ||
SQV <- [a, b]
],
[{unique, true}]),
qlc:q([
{X,Y} ||
X <- V1,
Y <- V2
],
[{unique, true}])
end
In the following example QLC V2 has been inserted to show the
joined generators and the join method chosen. A convention is
used for lookup join: the first generator (G2) is the one tra-
versed, the second (G1) is the table where constants are looked
up.
1> E1 = ets:new(e1, []),
E2 = ets:new(e2, []),
true = ets:insert(E1, [{1,a},{2,b}]),
true = ets:insert(E2, [{a,1},{b,2}]),
Q = qlc:q([{X,Z,W} ||
{X, Z} <- ets:table(E1),
{W, Y} <- ets:table(E2),
X =:= Y]),
io:format("~s~n", [qlc:info(Q)]).
begin
V1 =
qlc:q([
P0 ||
P0 = {W, Y} <-
ets:table(#Ref<0.3098908599.2283929601.256549>)
]),
V2 =
qlc:q([
[G1 | G2] ||
G2 <- V1,
G1 <-
ets:table(#Ref<0.3098908599.2283929601.256548>),
element(2, G1) =:= element(1, G2)
],
[{join, lookup}]),
qlc:q([
{X, Z, W} ||
[{X, Z} | {W, Y}] <- V2
])
end
keysort(KeyPos, QH1) -> QH2
keysort(KeyPos, QH1, SortOptions) -> QH2
Types:
KeyPos = key_pos()
SortOptions = sort_options()
QH1 = query_handle_or_list()
QH2 = query_handle()
Returns a query handle. When evaluating query handle QH2, the
answers to query handle QH1 are sorted by file_sorter:keysort/4
according to the options.
The sorter uses temporary files only if QH1 does not evaluate to
a list and the size of the binary representation of the answers
exceeds Size bytes, where Size is the value of option size.
keysort(KeyPos, QH1) is equivalent to keysort(KeyPos, QH1, []).
next_answers(QueryCursor) -> Answers | Error
next_answers(QueryCursor, NumberOfAnswers) -> Answers | Error
Types:
QueryCursor = query_cursor()
Answers = answers()
NumberOfAnswers = all_remaining | integer() >= 1
Error = {error, module(), Reason}
Reason = file_sorter:reason()
Returns some or all of the remaining answers to a query cursor.
Only the owner of QueryCursor can retrieve answers.
Optional argument NumberOfAnswers determines the maximum number
of answers returned. Defaults to 10. If less than the requested
number of answers is returned, subsequent calls to next_answers
return [].
q(QLC) -> QH
q(QLC, Options) -> QH
Types:
QH = query_handle()
Options = [Option] | Option
Option =
{max_lookup, MaxLookup} |
{cache, cache()} |
cache |
{join, Join} |
{lookup, Lookup} |
{unique, boolean()} |
unique
MaxLookup = integer() >= 0 | infinity
Join = any | lookup | merge | nested_loop
Lookup = boolean() | any
QLC = query_list_comprehension()
Returns a query handle for a QLC. The QLC must be the first ar-
gument to this function, otherwise it is evaluated as an ordi-
nary list comprehension. It is also necessary to add the follow-
ing line to the source code:
-include_lib("stdlib/include/qlc.hrl").
This causes a parse transform to substitute a fun for the QLC.
The (compiled) fun is called when the query handle is evaluated.
When calling qlc:q/1,2 from the Erlang shell, the parse trans-
form is automatically called. When this occurs, the fun substi-
tuted for the QLC is not compiled but is evaluated by
erl_eval(3erl). This is also true when expressions are evaluated
by file:eval/1,2 or in the debugger.
To be explicit, this does not work:
...
A = [X || {X} <- [{1},{2}]],
QH = qlc:q(A),
...
Variable A is bound to the evaluated value of the list compre-
hension ([1,2]). The compiler complains with an error message
("argument is not a query list comprehension"); the shell
process stops with a badarg reason.
q(QLC) is equivalent to q(QLC, []).
Options:
* Option {cache, ets} can be used to cache the answers to a
QLC. The answers are stored in one ETS table for each cached
QLC. When a cached QLC is evaluated again, answers are
fetched from the table without any further computations.
Therefore, when all answers to a cached QLC have been found,
the ETS tables used for caching answers to the qualifiers of
the QLC can be emptied. Option cache is equivalent to
{cache, ets}.
* Option {cache, list} can be used to cache the answers to a
QLC like {cache, ets}. The difference is that the answers
are kept in a list (on the process heap). If the answers
would occupy more than a certain amount of RAM memory, a
temporary file is used for storing the answers. Option
max_list_size sets the limit in bytes and the temporary file
is put on the directory set by option tmpdir.
Option cache has no effect if it is known that the QLC is to
be evaluated at most once. This is always true for the top-
most QLC and also for the list expression of the first gen-
erator in a list of qualifiers. Notice that in the presence
of side effects in filters or callback functions, the an-
swers to QLCs can be affected by option cache.
* Option {unique, true} can be used to remove duplicate an-
swers to a QLC. The unique answers are stored in one ETS ta-
ble for each QLC. The table is emptied every time it is
known that there are no more answers to the QLC. Option
unique is equivalent to {unique, true}. If option unique is
combined with option {cache, ets}, two ETS tables are used,
but the full answers are stored in one table only. If option
unique is combined with option {cache, list}, the answers
are sorted twice using keysort/3; once to remove duplicates
and once to restore the order.
Options cache and unique apply not only to the QLC itself but
also to the results of looking up constants, running match spec-
ifications, and joining handles.
Example:
In the following example the cached results of the merge join
are traversed for each value of A. Notice that without option
cache the join would have been carried out three times, once for
each value of A.
1> Q = qlc:q([{A,X,Z,W} ||
A <- [a,b,c],
{X,Z} <- [{a,1},{b,4},{c,6}],
{W,Y} <- [{2,a},{3,b},{4,c}],
X =:= Y],
{cache, list}),
io:format("~s~n", [qlc:info(Q)]).
begin
V1 =
qlc:q([
P0 ||
P0 = {X, Z} <-
qlc:keysort(1, [{a, 1}, {b, 4}, {c, 6}], [])
]),
V2 =
qlc:q([
P0 ||
P0 = {W, Y} <-
qlc:keysort(2, [{2, a}, {3, b}, {4, c}], [])
]),
V3 =
qlc:q([
[G1 | G2] ||
G1 <- V1,
G2 <- V2,
element(1, G1) == element(2, G2)
],
[{join, merge}, {cache, list}]),
qlc:q([
{A, X, Z, W} ||
A <- [a, b, c],
[{X, Z} | {W, Y}] <- V3,
X =:= Y
])
end
sort/1,2 and keysort/2,3 can also be used for caching answers
and for removing duplicates. When sorting answers are cached in
a list, possibly stored on a temporary file, and no ETS tables
are used.
Sometimes (see table/2) traversal of tables can be done by look-
ing up key values, which is assumed to be fast. Under certain
(rare) circumstances there can be too many key values to look
up. Option {max_lookup, MaxLookup} can then be used to limit the
number of lookups: if more than MaxLookup lookups would be re-
quired, no lookups are done but the table is traversed instead.
Defaults to infinity, which means that there is no limit on the
number of keys to look up.
Example:
In the following example, using the gb_table module from section
Implementing a QLC Table, there are six keys to look up: {1,a},
{1,b}, {1,c}, {2,a}, {2,b}, and {2,c}. The reason is that the
two elements of key {X, Y} are compared separately.
1> T = gb_trees:empty(),
QH = qlc:q([X || {{X,Y},_} <- gb_table:table(T),
((X == 1) or (X == 2)) andalso
((Y == a) or (Y == b) or (Y == c))]),
io:format("~s~n", [qlc:info(QH)]).
ets:match_spec_run(
lists:flatmap(fun(K) ->
case
gb_trees:lookup(K,
gb_trees:from_orddict([]))
of
{value, V} ->
[{K, V}];
none ->
[]
end
end,
[{1, a},
{1, b},
{1, c},
{2, a},
{2, b},
{2, c}]),
ets:match_spec_compile([{{{'$1', '$2'}, '_'},
[],
['$1']}]))
Options:
* Option {lookup, true} can be used to ensure that the qlc
module looks up constants in some QLC table. If there are
more than one QLC table among the list expressions of the
generators, constants must be looked up in at least one of
the tables. The evaluation of the query fails if there are
no constants to look up. This option is useful when it would
be unacceptable to traverse all objects in some table. Set-
ting option lookup to false ensures that no constants are
looked up ({max_lookup, 0} has the same effect). Defaults to
any, which means that constants are looked up whenever pos-
sible.
* Option {join, Join} can be used to ensure that a certain
join method is used:
* {join, lookup} invokes the lookup join method.
* {join, merge} invokes the merge join method.
* {join, nested_loop} invokes the method of matching every
pair of objects from two handles. This method is mostly
very slow.
The evaluation of the query fails if the qlc module cannot
carry out the chosen join method. Defaults to any, which
means that some fast join method is used if possible.
sort(QH1) -> QH2
sort(QH1, SortOptions) -> QH2
Types:
SortOptions = sort_options()
QH1 = query_handle_or_list()
QH2 = query_handle()
Returns a query handle. When evaluating query handle QH2, the
answers to query handle QH1 are sorted by file_sorter:sort/3 ac-
cording to the options.
The sorter uses temporary files only if QH1 does not evaluate to
a list and the size of the binary representation of the answers
exceeds Size bytes, where Size is the value of option size.
sort(QH1) is equivalent to sort(QH1, []).
string_to_handle(QueryString) -> QH | Error
string_to_handle(QueryString, Options) -> QH | Error
string_to_handle(QueryString, Options, Bindings) -> QH | Error
Types:
QueryString = string()
Options = [Option] | Option
Option =
{max_lookup, MaxLookup} |
{cache, cache()} |
cache |
{join, Join} |
{lookup, Lookup} |
{unique, boolean()} |
unique
MaxLookup = integer() >= 0 | infinity
Join = any | lookup | merge | nested_loop
Lookup = boolean() | any
Bindings = erl_eval:binding_struct()
QH = query_handle()
Error = {error, module(), Reason}
Reason = erl_parse:error_info() | erl_scan:error_info()
A string version of q/1,2. When the query handle is evaluated,
the fun created by the parse transform is interpreted by
erl_eval(3erl). The query string is to be one single QLC termi-
nated by a period.
Example:
1> L = [1,2,3],
Bs = erl_eval:add_binding('L', L, erl_eval:new_bindings()),
QH = qlc:string_to_handle("[X+1 || X <- L].", [], Bs),
qlc:eval(QH).
[2,3,4]
string_to_handle(QueryString) is equivalent to string_to_han-
dle(QueryString, []).
string_to_handle(QueryString, Options) is equivalent to
string_to_handle(QueryString, Options, erl_eval:new_bindings()).
This function is probably mainly useful when called from outside
of Erlang, for example from a driver written in C.
table(TraverseFun, Options) -> QH
Types:
TraverseFun = TraverseFun0 | TraverseFun1
TraverseFun0 = fun(() -> TraverseResult)
TraverseFun1 = fun((match_expression()) -> TraverseResult)
TraverseResult = Objects | term()
Objects = [] | [term() | ObjectList]
ObjectList = TraverseFun0 | Objects
Options = [Option] | Option
Option =
{format_fun, FormatFun} |
{info_fun, InfoFun} |
{lookup_fun, LookupFun} |
{parent_fun, ParentFun} |
{post_fun, PostFun} |
{pre_fun, PreFun} |
{key_equality, KeyComparison}
FormatFun = undefined | fun((SelectedObjects) -> Format-
edTable)
SelectedObjects =
all |
{all, NElements, DepthFun} |
{match_spec, match_expression()} |
{lookup, Position, Keys} |
{lookup, Position, Keys, NElements, DepthFun}
NElements = infinity | integer() >= 1
DepthFun = fun((term()) -> term())
FormatedTable = {Mod, Fun, Args} | abstract_expr() | string()
InfoFun = undefined | fun((InfoTag) -> InfoValue)
InfoTag = indices | is_unique_objects | keypos | num_of_ob-
jects
InfoValue = undefined | term()
LookupFun = undefined | fun((Position, Keys) -> LookupResult)
LookupResult = [term()] | term()
ParentFun = undefined | fun(() -> ParentFunValue)
PostFun = undefined | fun(() -> term())
PreFun = undefined | fun((PreArgs) -> term())
PreArgs = [PreArg]
PreArg = {parent_value, ParentFunValue} | {stop_fun, StopFun}
ParentFunValue = undefined | term()
StopFun = undefined | fun(() -> term())
KeyComparison = '=:=' | '=='
Position = integer() >= 1
Keys = [term()]
Mod = Fun = atom()
Args = [term()]
QH = query_handle()
Returns a query handle for a QLC table. In Erlang/OTP there is
support for ETS, Dets, and Mnesia tables, but many other data
structures can be turned into QLC tables. This is accomplished
by letting function(s) in the module implementing the data
structure create a query handle by calling qlc:table/2. The dif-
ferent ways to traverse the table and properties of the table
are handled by callback functions provided as options to qlc:ta-
ble/2.
* Callback function TraverseFun is used for traversing the ta-
ble. It is to return a list of objects terminated by either
[] or a nullary fun to be used for traversing the not yet
traversed objects of the table. Any other return value is
immediately returned as value of the query evaluation. Unary
TraverseFuns are to accept a match specification as argu-
ment. The match specification is created by the parse trans-
form by analyzing the pattern of the generator calling
qlc:table/2 and filters using variables introduced in the
pattern. If the parse transform cannot find a match specifi-
cation equivalent to the pattern and filters, TraverseFun is
called with a match specification returning every object.
* Modules that can use match specifications for optimized
traversal of tables are to call qlc:table/2 with an unary
TraverseFun. An example is ets:table/2.
* Other modules can provide a nullary TraverseFun. An exam-
ple is gb_table:table/1 in section Implementing a QLC Ta-
ble.
* Unary callback function PreFun is called once before the ta-
ble is read for the first time. If the call fails, the query
evaluation fails.
Argument PreArgs is a list of tagged values. There are two
tags, parent_value and stop_fun, used by Mnesia for managing
transactions.
* The value of parent_value is the value returned by Parent-
Fun, or undefined if there is no ParentFun. ParentFun is
called once just before the call of PreFun in the context
of the process calling eval/1,2, fold/3,4, or cursor/1,2.
* The value of stop_fun is a nullary fun that deletes the
cursor if called from the parent, or undefined if there is
no cursor.
* Nullary callback function PostFun is called once after the
table was last read. The return value, which is caught, is
ignored. If PreFun has been called for a table, PostFun is
guaranteed to be called for that table, even if the evalua-
tion of the query fails for some reason.
The pre (post) functions for different tables are evaluated
in unspecified order.
Other table access than reading, such as calling InfoFun, is
assumed to be OK at any time.
* Binary callback function LookupFun is used for looking up
objects in the table. The first argument Position is the key
position or an indexed position and the second argument Keys
is a sorted list of unique values. The return value is to be
a list of all objects (tuples), such that the element at Po-
sition is a member of Keys. Any other return value is imme-
diately returned as value of the query evaluation. LookupFun
is called instead of traversing the table if the parse
transform at compile time can determine that the filters
match and compare the element at Position in such a way that
only Keys need to be looked up to find all potential an-
swers.
The key position is obtained by calling InfoFun(keypos) and
the indexed positions by calling InfoFun(indices). If the
key position can be used for lookup, it is always chosen,
otherwise the indexed position requiring the least number of
lookups is chosen. If there is a tie between two indexed po-
sitions, the one occurring first in the list returned by In-
foFun is chosen. Positions requiring more than max_lookup
lookups are ignored.
* Unary callback function InfoFun is to return information
about the table. undefined is to be returned if the value of
some tag is unknown:
indices:
Returns a list of indexed positions, a list of positive
integers.
is_unique_objects:
Returns true if the objects returned by TraverseFun are
unique.
keypos:
Returns the position of the table key, a positive integer.
is_sorted_key:
Returns true if the objects returned by TraverseFun are
sorted on the key.
num_of_objects:
Returns the number of objects in the table, a non-negative
integer.
* Unary callback function FormatFun is used by info/1,2 for
displaying the call that created the query handle of the ta-
ble. Defaults to undefined, which means that info/1,2 dis-
plays a call to '$MOD':'$FUN'/0. It is up to FormatFun to
present the selected objects of the table in a suitable way.
However, if a character list is chosen for presentation, it
must be an Erlang expression that can be scanned and parsed
(a trailing dot is added by info/1,2 though).
FormatFun is called with an argument that describes the se-
lected objects based on optimizations done as a result of
analyzing the filters of the QLC where the call to qlc:ta-
ble/2 occurs. The argument can have the following values:
{lookup, Position, Keys, NElements, DepthFun}.:
LookupFun is used for looking up objects in the table.
{match_spec, MatchExpression}:
No way of finding all possible answers by looking up keys
was found, but the filters could be transformed into a
match specification. All answers are found by calling Tra-
verseFun(MatchExpression).
{all, NElements, DepthFun}:
No optimization was found. A match specification matching
all objects is used if TraverseFun is unary.
NElements is the value of the info/1,2 option n_elements.
DepthFun is a function that can be used for limiting the
size of terms; calling DepthFun(Term) substitutes '...'
for parts of Term below the depth specified by the
info/1,2 option depth.
If calling FormatFun with an argument including NElements
and DepthFun fails, FormatFun is called once again with an
argument excluding NElements and DepthFun ({lookup, Posi-
tion, Keys} or all).
* The value of option key_equality is to be '=:=' if the table
considers two keys equal if they match, and to be '==' if
two keys are equal if they compare equal. Defaults to '=:='.
For the various options recognized by table/1,2 in respective
module, see ets(3erl), dets(3erl), and mnesia(3erl).
SEE ALSO
dets(3erl), erl_eval(3erl), erlang(3erl), error_logger(3erl),
ets(3erl), file(3erl), file_sorter(3erl), mnesia(3erl), shell(3erl),
Erlang Reference Manual, Programming Examples
Ericsson AB stdlib 3.13 qlc(3erl)