cuddl::detail::path_loaders Class Reference

Loader state a build fed by paths keeps for its whole run. More...

Public Member Functions
	path_loaders (cuda::stream_ref stream, std::span< std::filesystem::path const > paths, unsigned parser_workers, bool page_locked, bool in_place, uint32_t pack_k, bool direct_reads)
size_t	workers () const noexcept
	Loaders this build runs.
size_t	page_locked_buffers () const noexcept
	Page-locked buffers the loaders held, for the statistics.
Result< std::unique_ptr< fastx_sequence_file > >	take (std::span< std::filesystem::path const > paths, size_t id)
	Loads genome id, through the pool when the build runs more than one loader.
void	reset () noexcept
	Joins the loaders. They hold leases, so pool state is only readable after this.

Static Public Member Functions
static size_t	worker_count (size_t paths, unsigned parser_workers) noexcept
	Loaders to run: at most one per input, at most the requested workers, at most the hardware, and never none.

Detailed Description

Loader state a build fed by paths keeps for its whole run.

Reused decompression buffers avoid repeated allocations, and can be page-locked for DMA. The pool hands out load results a few files ahead of the consumer, which takes them in order. A window of one worker-worth leaves the consumer waiting on a straggler while every other loader sits idle.

Definition at line 1022 of file database_staging.cuh.

Constructor & Destructor Documentation

path_loaders()

cuddl::detail::path_loaders::path_loaders ( cuda::stream_ref  stream, std::span< std::filesystem::path const >  paths, unsigned  parser_workers, bool  page_locked, bool  in_place, uint32_t  pack_k, bool  direct_reads  )

inline

Parameters

stream	Stream owning the loader buffers.
paths	Genome files in reference-ID order.
parser_workers	Loader ceiling; the build clamps it to inputs and hardware.
page_locked	gives the loaders page-locked buffers to decompress into.
in_place	says the stager's kernels read the loaders' buffers themselves.
pack_k	packs FASTA into two-bit words for k-mers this long, a quarter of the bytes to stage, as pack_fastx_sequence_file does.
direct_reads	reads files with O_DIRECT, as read_whole_file does.

Definition at line 1032 of file database_staging.cuh.

        : workers_(worker_count(paths.size(), parser_workers)),
          buffers_(stream, workers_, size_t{32} << 20, page_locked, in_place),
          page_locked_(page_locked),
          pack_k_(pack_k) {
        // One buffer per in-flight file plus the stager's hold ring, plus headroom: a worker
        // asks for its next file while every loaded file still holds a lease. A pool smaller
        // than the window makes the remaining files allocate their own buffers.
        auto const window = workers_ * 4;
        buffers_.set_capacity(
            window + stager_hold_slots + 2, pack_k != 0 ? packed_buffer_bytes : 0
        );
        source_ = {acquire_sequence_target, &buffers_, direct_reads};
        if (workers_ > 1) pool_.emplace(paths, workers_, source_, window, false, false, pack_k);
    }

Here is the call graph for this function:

Member Function Documentation

page_locked_buffers()

size_t cuddl::detail::path_loaders::page_locked_buffers ( ) const

inlinenoexcept

Page-locked buffers the loaders held, for the statistics.

Definition at line 1064 of file database_staging.cuh.

                                                              {
        return page_locked_ ? buffers_.buffers() : 0;
    }

reset()

void cuddl::detail::path_loaders::reset ( )

inlinenoexcept

Joins the loaders. They hold leases, so pool state is only readable after this.

Definition at line 1078 of file database_staging.cuh.

                          {
        pool_.reset();
    }

take()

Result< std::unique_ptr< fastx_sequence_file > > cuddl::detail::path_loaders::take ( std::span< std::filesystem::path const >  paths, size_t  id  )

inline

Loads genome id, through the pool when the build runs more than one loader.

Definition at line 1070 of file database_staging.cuh.

                                                            {
        if (pool_.has_value()) return pool_->take(id);
        auto loaded = load_fastx_sequence_file(paths[id].string(), source_, false, false, pack_k_);
        if (pack_k_ != 0 && loaded) pack_fastx_sequence_file(**loaded, pack_k_);
        return loaded;
    }

Here is the call graph for this function:

worker_count()

static size_t cuddl::detail::path_loaders::worker_count ( size_t  paths, unsigned  parser_workers  )

inlinestaticnoexcept

Loaders to run: at most one per input, at most the requested workers, at most the hardware, and never none.

Definition at line 1084 of file database_staging.cuh.

                                                                               {
        auto const hardware = std::max(1U, std::thread::hardware_concurrency());
        auto const loaders = static_cast<size_t>(std::min(parser_workers, hardware));
        return std::max<size_t>(1, std::min(paths, loaders));
    }

Here is the caller graph for this function:

workers()

size_t cuddl::detail::path_loaders::workers ( ) const

inlinenoexcept

Loaders this build runs.

The pool allocates buffers lazily, independently of the number of loader threads.

Definition at line 1059 of file database_staging.cuh.

                                                  {
        return workers_;
    }

---

The documentation for this class was generated from the following file:

• database_staging.cuh