Data::Stream::Bulk(3) User Contributed Perl DocumentationData::Stream::Bulk(3)NAMEData::Stream::Bulk - N at a time iteration API
VERSION
version 0.11
SYNOPSIS
# get a bulk stream from somewere
my $s = Data::Stream::Bulk::Foo->new( ... );
# can be used like this:
until ( $s->is_done ) {
foreach my $item ( $s->items ) {
process($item);
}
}
# or like this:
while( my $block = $s->next ) {
foreach my $item ( @$block ) {
process($item);
}
}
DESCRIPTION
This module tries to find middle ground between one at a time and all
at once processing of data sets.
The purpose of this module is to avoid the overhead of implementing an
iterative api when this isn't necessary, without breaking forward
compatibility in case that becomes necessary later on.
The API optimizes for when a data set typically fits in memory and is
returned as an array, but the consumer cannot assume that the data set
is bounded.
The API is destructive in order to minimize the chance that resultsets
are leaked due to improper usage.
API
Required Methods
The API requires two methods to be implemented:
is_done
Should return true if the stream is exhausted.
As long as this method returns a false value (not done) "next"
could potentially return another block.
next
Returns the next block.
Note that "next" is not guaranteed to return an array reference,
even if "is_done" returned false prior to calling it.
Convenience Methods
items
This method calls "next" and dereferences the result if there are
pending items.
all Force evaluation of the entire resultset.
Note that for large data sets this might cause swap thrashing of
various other undesired effects. Use with caution.
cat @streams
Concatenates this stream with @streams, returning a single stream.
list_cat @tail
Returns a possibly cleaned up list of streams.
Used by "cat".
Overridden by Data::Stream::Bulk::Array, Data::Stream::Bulk::Cat
and Data::Stream::Bulk::Nil to implement some simple short
circuiting.
filter $filter
Applies a per-block block filter to the stream.
Returns a possibly new stream with the filtering layered.
$filter is invoked once per block and should return an array
reference to the filtered block.
chunk $chunk_size
Chunks the input stream so that each block returned by "next" will
have at least $chunk_size items.
loaded
Should be overridden to return true if all the items are already
realized (e.g. in the case of Data::Stream::Bulk::Array).
Returns false by default.
When true calling "all" is supposed to be safe (memory usage should
be in the same order of magnitude as stream's own usage).
This is typically useful when tranforming an array is easier than
transorming a stream (e.g. optional duplicate filtering).
CLASSES
Data::Stream::Bulk::Array
This class is not a stream at all, but just one block. When the
data set easily fits in memory this class can be used, while
retaining forward compatibility with larger data sets.
Data::Stream::Bulk::Callback
Callback driven iteration.
Data::Stream::Bulk::Chunked
Wrapper to return larger blocks from an existing stream.
Data::Stream::Bulk::DBI
Bulk fetching of data from DBI statement handles.
Data::Stream::Bulk::DBIC
DBIx::Class::ResultSet iteration.
Data::Stream::Bulk::FileHandle
Iterates over lines in a text file.
Data::Stream::Bulk::Nil
An empty result set.
Data::Stream::Bulk::Cat
A concatenation of several streams.
Data::Stream::Bulk::Filter
A filter wrapping a stream.
SEE ALSO
HOP::Stream, Iterator, Class::Iterator etc for one by one iteration
DBI, DBIx::Class::ResultSet
POE::Filter
Data::Page
Parallel::Iterator
<http://en.wikipedia.org/wiki/MapReduce>, LISP, and all that other kool
aid
TODO
Sorted streams
Add a hint for sorted streams (like "loaded" but as an attribute in
the base role).
Introduce a "merge" operation for merging of sorted streams.
Optimize "unique" to make use of sorting hints for constant space
uniquing.
More utility functions
To assist in proccessing and creating streams.
Coercion tables
Moose::Util::TypeConstraints
VERSION CONTROL
This module is maintained using git. You can get the latest version
from http://github.com/nothingmuch/data-stream-bulk/
<http://github.com/nothingmuch/data-stream-bulk/>.
AUTHOR
Yuval Kogman <nothingmuch@woobling.org>
COPYRIGHT AND LICENSE
This software is copyright (c) 2012 by Yuval Kogman.
This is free software; you can redistribute it and/or modify it under
the same terms as the Perl 5 programming language system itself.
perl v5.14.2 2012-02-14 Data::Stream::Bulk(3)
