cuddl::detail Namespace Reference

Namespaces
namespace	compact_impl
namespace	device_gzip

Classes
struct	bases_packer
	A base packer and the bases it takes per step, a multiple of sixteen up to 64. More...
struct	blacklist_view
	Exact membership within the candidate's destination bucket. More...
struct	cardinality_payload
	Per-thread accumulator feeding the CUB block reduction for cardinality. More...
struct	cardinality_scratch
	Device-visible scratch for the owning cardinality wrapper. More...
struct	database_file_reader
struct	database_file_writer
class	database_stager
	Encodes genome records into register rows through a device arena. More...
struct	decoded_fasta
struct	decompression_source
	Supplies storage for one file, or a null target when none is available. More...
struct	decompression_target
	Host-visible buffer a caller offers for decompressed bytes. More...
class	device_fasta_pipeline
	Normalises FASTA batches supplied compressed, uncompressed, or already inflated. More...
struct	fasta_parse_result
	Result of parsing one FASTA/FASTQ file into packed canonical k-mers. More...
class	fastx_fastq_reader
	FASTQ parser over a contiguous in-memory buffer. More...
class	fastx_load_pool
class	fastx_mapped_file
	Read-only contiguous file payload for in-memory FASTX parsing. More...
struct	fastx_packed_record
	One run of bases packed two bits each: its first word and its length in bases. More...
struct	fastx_sequence_extent
	A contiguous run of sequence bases (record headers and line breaks excluded). More...
struct	fastx_sequence_file
struct	fastx_sequence_span
	One parallel scan span of the combined FASTA sequence stream. More...
class	file_writer
	Writes a file from a thread of its own, so the caller produces the next bytes while earlier ones reach storage. More...
struct	gzip_file_probe
	What a gzip file's header and trailer say, or that a file is plain FASTA, read without inflating it. More...
struct	inflated_genome
	One genome the device inflated: its compacted bases, resident on the device. More...
class	io_ring
	Stand-in where the build has no io_uring headers: never available. More...
struct	packed_identity
	Weighted k-mer identity of one packed result, for the GPU conversion. More...
class	path_loaders
	Loader state a build fed by paths keeps for its whole run. More...
class	pinned_pool_trim
	Returns the free pages of a device_pinned_pool to the OS when it leaves scope. More...
struct	plane_counts
	Bit-sliced pairwise counts of one 32-bucket group, accumulated per lane. More...
struct	posting_bitmap_cell
	Selects posting lists whose bitmap uses no more storage than their reference IDs. More...
class	reference_database_view
	Non-owning, trivially copyable view of one selected reference-row backing. More...
struct	reference_delta_view
	Reference rows stored as a base reference plus the buckets that differ from it. More...
struct	reference_deltas
struct	reference_file_assembly
	Builds a reference_database_file from gathered parts; the file's members are private. More...
struct	reference_index_digest
struct	refine_cells
	How a refinement-style block splits a tile: cell c holds query group c % groups and the reference words from first_word + (c / groups) * block_words. More...
struct	sequence_batch_chunk
	One piece of sequence of one genome, and where its bytes are. More...
class	sequence_buffer_pool
struct	sequence_genome
	One genome as its records, in order. No records keeps the genome's ID with a zero row. More...
struct	sequence_record
	One genome of bases the caller already holds. More...
struct	shared_gzip_decompressor
	libdeflate handle, one per thread because the object is single-threaded. More...
struct	sketch_scratch
	Persistent zero-copy scratch used by owning synchronous result wrappers. More...
class	sketch_view
	Non-owning implementation view of a DDL sketch. More...
struct	span_result
	One worker's rolling-window state and output. More...
struct	sparse_index_key
struct	sparse_reference_id
struct	sparse_segment_offset
struct	staging_plan
	Sizing a build resolves before its first genome. More...
struct	summary_payload
	Per-thread accumulator feeding a CUB block reduction for fused summaries. More...
struct	top_k_row_offset
	Start of genome row i of a collector's heaps, rows of k slots. More...
struct	top_k_slots
	Each genome's k best neighbours so far: keys and packed counts in parallel rows of k, each row a min-heap of keys whose root is the row's worst key, zero while the row has room. More...
struct	word_popcount
	Popcount of one pass-bit word. More...
class	worker_pool
	Fixed threads that run indexed work with the caller, for work repeated per batch. More...

Typedefs
using	fastx_line_end_fn = decltype(&fastx_line_end_scalar)
using	device_scope = ::cuda::__ensure_current_context
	Makes stream's device current for the enclosing scope and restores the previous context on exit.

Enumerations
enum class	decompression_backend : uint8_t {   automatic , hybrid , cpu , gpu ,   gpu_cores }
	Where inputs are decompressed; sketch construction always runs on the GPU. More...
enum class	transfer_mode : uint8_t { automatic , in_place , pinned , staged }
	How a build gets CPU-parsed sequence bytes to the device. More...
enum class	fastx_format : uint8_t { unknown , fasta , fastq }
	Detected record format for a FASTX buffer. More...
enum class	hybrid_variant : uint8_t { bbtools , paper }
	Which HybridDDL blend the variant kernel computes. More...
enum class	refine_candidates { bitmap , all , reference_mask }
	Pairs refine_batch_bitmap_kernel compares. More...
enum class	pinned_placement : uint8_t { device_node , host }
	Where a device_pinned_pool takes its pages. More...

Functions
constexpr bool	ablated (std::string_view) noexcept
	Always false: this build compiles the ablation switches out.
__host__ __device__ double	minimum_mle (double bucket_count, double empty_count, double sum_restored) noexcept
	Estimates the number of distinct hashes from the minimum hash in each bucket.
__host__ __device__ double	mean_m (double bucket_count, double filled, double sum_restored) noexcept
	MeanM estimate retained as the full-sketch form of minimum_mle.
__host__ __device__ double	cardinality (double bucket_count, double empty_count, double sum_restored) noexcept
	Public DDL cardinality estimate.
__host__ __device__ float	minimum_mle_f32 (float bucket_count, float empty_count, float sum_restored) noexcept
	Single-precision variants of the cardinality estimators.
__host__ __device__ float	mean_m_f32 (float bucket_count, float filled, float sum_restored) noexcept
	Single-precision mean_m for the throughput-bound cardinality reduction.
__host__ __device__ float	cardinality_f32 (float bucket_count, float empty_count, float sum_restored) noexcept
	Single-precision cardinality for the throughput-bound cardinality reduction.
__device__ void	classify (pairwise_counts &target, uint32_t this_score, uint32_t other_score) noexcept
	Classifies one score relative to this into target.
template<size_t BucketCount, typename Layout = default_register_layout>
__host__ Result< void >	launch_construction (device_span< uint64_t const > input, device_span< uint32_t > registers, cuda::stream_ref stream, cuda::std::optional< blacklist_view > blacklist=cuda::std::nullopt)
	Launches construction of a sketch through a CTA-local shared register array merged into registers (add_shared_kernel).
bool	device_reads_pageable_memory (cuda::device_ref device) noexcept
	Whether device reads pageable host memory itself.
bool	stages_in_place (transfer_mode mode, cuda::device_ref device) noexcept
	Whether a build in mode stages the caller's bytes in place.
bool	pages_locked (transfer_mode mode, cuda::device_ref device) noexcept
	Whether a build in mode decompresses into page-locked buffers.
Result< size_t >	available_device_bytes (cuda::stream_ref stream)
	Device bytes an allocation on stream can still get.
template<uint32_t K, size_t BucketCount>
Result< staging_plan >	plan_staging (size_t references, uint64_t staged_ceiling, std::optional< size_t > staging_bytes, cuda::stream_ref stream)
	Sizes the arena and the row store from free device memory and the corpus.
template<size_t BucketCount>
Result< void >	download_store (device_span< uint32_t const > store, std::span< uint16_t > scores, cuda::stream_ref stream)
	Copies the winner scores of device store rows into host score rows.
decompression_target	acquire_sequence_target (void *context, size_t bytes)
	decompression_source trampoline for sequence_buffer_pool.
template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout, typename Fill , typename Sink >
Result< void >	stage_groups (size_t genomes, uint64_t staged_ceiling, std::optional< size_t > staging_bytes, cuda::stream_ref stream, reference_build_statistics *statistics, Fill &&fill, Sink &&sink, bool in_place=false, cuda::std::optional< blacklist_view > blacklist=cuda::std::nullopt)
	Runs the staging loop over a collection, filling one group at a time.
template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout, typename Sink >
Result< void >	stage_paths (std::span< std::filesystem::path const > paths, cuda::stream_ref stream, std::optional< size_t > staging_bytes, unsigned parser_workers, transfer_mode transfer, decompression_backend decompression, bool direct_reads, reference_build_statistics *statistics, Sink &&sink, cuda::std::optional< blacklist_view > blacklist=cuda::std::nullopt)
	Stages a path collection, handing each group's device rows to sink.
template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout, typename Sink >
Result< void >	stage_sequences (std::span< sequence_genome const > genomes, cuda::stream_ref stream, std::optional< size_t > staging_bytes, reference_build_statistics *statistics, Sink &&sink, cuda::std::optional< blacklist_view > blacklist=cuda::std::nullopt)
	Stages bases the caller already holds, handing each group's device rows to sink.
uint32_t	bgzf_block_size (unsigned char const *header) noexcept
	A BGZF block's total size from its 18-byte header, or 0 if it is not one: a gzip header with only FEXTRA set, holding just the BC subfield.
uint32_t	crc32_combine (uint32_t a, uint32_t b, uint64_t length) noexcept
	CRC32 of A followed by B, from CRC32(A), CRC32(B) and B's length, as zlib's crc32_combine: CRC32(A) times x^(8 * length) modulo the reflected polynomial, plus CRC32(B).
template<typename Block >
std::optional< std::pair< uint32_t, uint64_t > >	walk_bgzf (unsigned char const *bytes, uint64_t size, Block &&block)
	Calls block(deflate_at, deflate_bytes, inflated) for each non-empty block of the BGZF file in bytes.
gzip_file_probe	probe_gzip_file (std::filesystem::path const &path)
	Reads the header and trailer of path, and of every block of a BGZF file.
template<typename Work >
void	parallel_for (size_t count, size_t workers, Work &&work)
	Runs work(index) for every index below count on workers threads.
bool	has_gzip_signature_portable (char const *data, size_t size)
	True when data holds what could start another gzip member: 1f 8b 08, then a flag byte with its reserved bits clear.
auto	resolve_gzip_signature_scanner () -> bool(*)(char const *, size_t)
	Selects the widest signature scanner the host supports.
constexpr uint8_t	encode_base (char base) noexcept
	Maps one DNA byte to its 2-bit symbol (A=0, C=1, T=2, G=3) or 0xff when invalid.
constexpr uint64_t	reverse_complement (uint64_t packed, uint32_t k) noexcept
	Reverse-complements a packed k-mer (2 bits per base) with bit-parallel reversal.
std::vector< fastx_sequence_span >	fastx_split_sequence_spans (std::vector< fastx_sequence_extent > const &extents, uint32_t seed_bases, uint32_t count)
	Splits the combined FASTA sequence stream into count contiguous spans.
void	consume_byte (char ch, uint32_t k, uint64_t mask, span_result &result, uint64_t &window, uint32_t &window_len)
	Feeds one byte of sequence through the DDL rolling-window extraction.
Result< fasta_parse_result >	parse_fasta (std::string const &path, uint32_t k, unsigned threads=std::thread::hardware_concurrency())
	Parses a FASTA/FASTQ file's sequence bases into packed canonical k-mers of length k.
constexpr bool	fastx_is_sequence_whitespace (char ch) noexcept
	True for bytes that FASTA sequence consumers conventionally skip between bases.
size_t	fastx_line_end_scalar (std::string_view data, size_t position)
fastx_line_end_fn	resolve_fastx_line_end ()
	Selects the fastest line-end scanner the host supports.
size_t	fastx_line_end (std::string_view data, size_t position)
	First line-end position at or after position using the resolved scanner.
uint32_t	pack_bases16_portable (char const *bases, size_t count, uint32_t &word)
	Packs up to sixteen bases into a two-bit word, base j in bits 2j and 2j+1.
uint64_t	pack_bases_pieces (char const *bases, size_t count, uint32_t *words, size_t width)
	Packs width bases from sixteen-base pieces, for a tail shorter than a full step.
uint64_t	pack_bases_portable (char const *bases, size_t count, uint32_t *words)
bases_packer	resolve_bases_packer ()
	Selects the widest base packer the host supports.
void	pack_fastx_sequence_file (fastx_sequence_file &result, uint32_t k)
	Packs a parsed file's records in place into two-bit words, run by run.
libdeflate_result	decompress_gzip_member (libdeflate_decompressor *decoder, std::string_view input, char *output, size_t capacity, size_t &consumed, size_t &produced)
	Decodes one gzip member, checking structure and ISIZE but not checksums.
Result< void >	gunzip_members_into (std::string_view input, std::string &output, std::string const &path)
	Decompresses gzip members from a mapped buffer into output, appending.
Result< size_t >	gunzip_members_direct (std::string_view input, decompression_target target, bool &insufficient, std::string const &path)
	Decompresses gzip members straight into a caller's buffer.
uint64_t	gzip_decompressed_size (std::string const &path)
	Uncompressed size the gzip trailer advertises, or 0 when it cannot be read.
size_t	gzip_size_hint (std::string_view input)
char *	compact_sequence_whitespace (char const *first, char const *last, char *out)
	Copies bases from [first, last) to out, dropping sequence whitespace.
char const *	fasta_next_header (char const *first, char const *last)
	First '>' in [first, last) that starts a line, or last.
char *	fasta_sequence_compact (char const *first, char const *last, char *out, char const *&stop)
	Compacts one FASTA record's sequence from first to out, stopping at the next header.
std::vector< fastx_sequence_extent >	fastx_fasta_compact_extents (std::string_view data, char *out)
	Collects FASTA sequence extents with whitespace compacted out, in one pass over data.
void	compact_fastx_sequence_extents (fastx_sequence_file &result)
	Compacts sequence whitespace in place, leaving every extent a run of bases.
Result< void >	parse_fastx_sequence_file (fastx_sequence_file &result, std::string const &path)
Result< void >	pack_fasta_chunks (fastx_sequence_file &result, std::string_view data, char *out, uint32_t k, std::string const &path)
	Compact and pack one cache-sized block at a time, without a genome-sized ASCII write.
Result< std::unique_ptr< fastx_sequence_file > >	finish_fastx_sequence_file (std::unique_ptr< fastx_sequence_file > result, std::string_view data, std::string const &path, bool defer_fasta, uint32_t pack_k)
	Parses, packs or defers the bytes data of a loaded file, which result holds.
Result< std::optional< std::string_view > >	inflate_and_pack (fastx_sequence_file &result, std::string_view input, std::string const &path, decompression_source source, uint32_t k)
	Inflates gzip input in this thread's reusable heap buffer and, for FASTA, packs it there and moves only the packed words into a buffer from source.
Result< std::unique_ptr< fastx_sequence_file > >	decode_fastx_sequence_file (std::unique_ptr< fastx_mapped_file > file, std::string const &path, decompression_source source, bool defer_fasta, uint32_t pack_k=0)
Result< std::unique_ptr< fastx_sequence_file > >	load_fastx_sequence_file (std::string const &path, decompression_source source={}, bool defer_fasta=false, bool read=false, uint32_t pack_k=0)
__host__ __device__ constexpr uint64_t	splitmix64 (uint64_t value) noexcept
	SplitMix64 finalizer, applied to a packed k-mer XOR the domain seed.
__host__ __device__ constexpr uint64_t	hash_kmer (uint64_t packed_kmer) noexcept
	Hashes a canonical packed k-mer into the value space used by DDL registers.
template<size_t BucketCount>
__host__ __device__ constexpr size_t	bucket_of (uint64_t hash) noexcept
	Selects the register bucket for hash given a power-of-two BucketCount.
constexpr __host__ __device__ __forceinline__ uint64_t	rotl64 (uint64_t x, int8_t r)
	Rotates x left by r bits.
template<typename T >
__host__ __device__ __forceinline__ T	load_chunk (const uint8_t *data, uint64_t index)
	Loads a chunk of type T from data at byte offset index*sizeof(T).
constexpr __host__ __device__ __forceinline__ uint64_t	finalize (uint64_t h)
	Applies the xxHash-64 final mixing (avalanche) step.
__host__ __device__ uint64_t	xxhash64 (const uint8_t *bytes, uint64_t size, uint64_t seed)
	Computes xxHash-64 over a byte range without allocating or copying it.
template<typename T >
__host__ __device__ uint64_t	xxhash64 (const T &key, uint64_t seed=0)
	Computes xxHash-64 over the raw bytes of a value.
__device__ double	interpolate_mean_m_cf (double estimate) noexcept
	Interpolates the MeanM correction factor for estimate from the calibration table.
__device__ double	lc_min (uint32_t const *bins, double bucket_count) noexcept
	Linear-counting estimate from the first occupied NLZ tier.
__device__ double	dlc (uint32_t const *bins, double bucket_count, double lc) noexcept
	Distributed linear-counting estimate, blended from per-tier estimates by precision.
__device__ double	hybrid (double low_estimate, double zone_estimate, double corrected_mean, double bucket_count) noexcept
	Log-blended low and corrected estimates across the hybrid transition zone.
__device__ double	bbtools_mean_m (double bucket_count, double filled, double restored_sum) noexcept
	BBTools occupancy-corrected MeanM estimate from restored magnitudes.
__device__ hybrid_cardinality_estimates	hybrid_estimates (uint32_t const *bins, double bucket_count, double restored_sum) noexcept
	Fused BBTools, paper, linear-counting, DLC, and raw MeanM estimates.
__device__ float	interpolate_mean_m_cf_f32 (float estimate) noexcept
	Single-precision variant of the hybrid estimator tail.
__device__ float	lc_min_f32 (uint32_t const *bins, float bucket_count) noexcept
	Single-precision lc_min for the latency-bound hybrid scan.
__device__ float	dlc_f32 (uint32_t const *bins, float bucket_count, float lc) noexcept
	Single-precision dlc for the latency-bound hybrid scan.
__device__ float	hybrid_f32 (float low_estimate, float zone_estimate, float corrected_mean, float bucket_count) noexcept
	Single-precision hybrid for the latency-bound hybrid scan.
__device__ float	bbtools_mean_m_f32 (float bucket_count, float filled, float restored_sum) noexcept
	Single-precision bbtools_mean_m for the latency-bound hybrid scan.
__device__ hybrid_cardinality_estimates	hybrid_estimates_f32 (uint32_t const *bins, float bucket_count, float restored_sum) noexcept
	Single-precision hybrid_estimates for the latency-bound hybrid scan.
template<size_t BucketCount, typename Layout = default_register_layout, bool HasBlacklist = false>
__global__	__launch_bounds__ (shared_construction_block_size) void add_shared_kernel(uint64_t const *input
	Constructs a sketch through a CTA-local shared-memory register array.
__device__ uint2	lane_posting_range (uint32_t const *offsets, uint16_t const *sorted_keys, uint32_t reference_count, uint32_t bucket, uint32_t key, uint32_t key_count)
	One thread's posting range for a bucket/key cell of a dense or sparse index.
__host__ __device__ constexpr uint32_t	sparse_directory_shift (uint32_t reference_count) noexcept
	Low key bits a sparse key directory cell spans for reference_count references.
__host__ __device__ constexpr uint32_t	sparse_directory_entries (uint32_t reference_count) noexcept
	Bounds per bucket in a sparse key directory.
__global__	__launch_bounds__ (block_size) static void build_sparse_directory_kernel(uint16_t const *sorted_keys
	Builds a sparse key directory: entry c of bucket b is the first position in b's sorted keys whose key is at least c << shift.
	for (auto i=static_cast< size_t >(blockIdx.x) *block_size+threadIdx.x;i< total;i+=static_cast< size_t >(gridDim.x) *block_size)
	if (local !=0U)
__device__ uint2	directory_posting_range (uint16_t const *sorted_keys, uint32_t const *directory, uint32_t reference_count, uint32_t bucket, uint32_t sparse_key)
	One sparse bucket/key posting range, narrowed through the bucket's key directory.
template<size_t BucketCount, typename QueryRow >
__global__	__launch_bounds__ (block_size) void count_batch_index_matches_kernel(QueryRow const *queries
	Counts index matches for every query/reference pair in one tile.
bool	uses_tiled_index_counts (uint32_t reference_count, uint32_t query_count, uint32_t indexed_bucket_count) noexcept
	Whether a batch counts through count_batch_index_tile_kernel.
uint32_t	index_tile_references (uint32_t reference_count, uint32_t query_count, uint32_t multiprocessors, uint32_t max_shared_bytes) noexcept
	References per tile, bounded by the device's shared memory and small enough that the batch launches about four blocks per SM.
template<size_t BucketCount>
__global__ void	hybrid_posting_ranges_kernel (uint16_t const *queries, uint32_t const *offsets, uint32_t const *bitmap_cells, uint32_t bitmap_count, uint32_t words, uint32_t indexed_buckets, uint16_t key_mask, uint2 *ranges, uint32_t *bitmap_starts, uint32_t *bitmap_counts)
	Splits each query's posting ranges into short lists and bitmap offsets.
template<uint32_t CounterBits>
__device__ uint32_t	sum_posting_bitmaps (uint32_t const *bitmaps, uint32_t const *starts, uint32_t count, uint32_t word, uint32_t minimum_matches, uint32_t passed, uint32_t(&counters)[CounterBits])
	Sums one query's posting bitmaps over one word of references in bit-sliced counters.
template<size_t BucketCount, uint32_t CounterBits, typename QueryRow >
__global__	__maxnreg__ (48) void count_batch_index_tile_kernel(QueryRow const *queries
	Counts index matches for one query against one reference tile in shared memory.
	if (candidate_bits !=nullptr &&all_to_all &&high<=query_id_offset+query_index+1U)
if	constexpr (CounterBits !=0U)
	__syncthreads ()
	if (candidate_bits==nullptr)
template<typename ReferenceScore >
__device__ __forceinline__ void	classify_query_words (pairwise_counts &target, uint32_t const (&q)[8], ReferenceScore const *reference) noexcept
	Classifies 16 buckets of an already loaded 16-bit query chunk against one load_256_alignment aligned reference chunk.
template<typename SearchResult >
__device__ void	refine_write_result (uint32_t index, uint32_t query_index, uint32_t reference_id, uint32_t query_id_offset, uint32_t const *match_counts, uint32_t pair_id, pairwise_counts local, uint32_t lane, SearchResult *results, uint32_t *result_match_counts)
	Warp-reduces one refined candidate and writes its stable result slot.
__host__ __device__ constexpr uint32_t	score_plane_index (uint32_t group, uint32_t plane) noexcept
	Position of plane plane of 32-bucket group group within a row's bit-planes.
__device__ __forceinline__ void	write_score_planes (uint32_t score, uint32_t group, uint32_t lane, uint32_t *planes)
	Writes the bit-planes of one 32-bucket group whose scores the warp holds, one per lane.
template<size_t BucketCount>
__global__	__launch_bounds__ (block_size) void build_score_planes_kernel(uint16_t const *rows
	Transposes compact score rows into the bit-plane layout of score_plane_index.
__device__ __forceinline__ void	load_plane_group (uint4 const *row, uint32_t group, uint32_t(&words)[score_planes])
	Loads group group of one bit-plane row (score_plane_index layout).
__device__ __forceinline__ uint32_t	plane_any (uint32_t const (&planes)[score_planes]) noexcept
	Buckets of a plane group holding a nonzero score.
__device__ __forceinline__ uint32_t	plane_low_any (uint32_t const (&planes)[score_planes]) noexcept
	Buckets of a plane group whose score is nonzero below the top plane.
template<bool CountMatches, bool ReferenceFull = false>
__device__ __forceinline__ void	compare_plane_group (uint32_t const (&q)[score_planes], uint32_t const (&r)[score_planes], uint32_t reference_any, uint32_t reference_low_any, uint16_t key_mask, uint32_t match_lanes, plane_counts &counts) noexcept
	Adds one group's comparison of query planes q with reference planes r, whose nonzero buckets reference_any the caller hoists out of per-query loops.
uint32_t	warp_grid_blocks (size_t warps) noexcept
	Blocks for a grid-stride launch covering warps warps of block_size threads.
__device__ __forceinline__ uint32_t	next_cell (uint32_t *cell_counter, uint32_t cell, uint32_t &claimed)
	The cell a refinement-style block processes after cell: claimed from cell_counter, a zeroed per-launch counter, so blocks balance uneven cells; without one, grid-stride.
template<bool UpperTriangle>
__device__ __forceinline__ refine_cells	make_refine_cells (uint32_t query_id_offset, uint32_t query_count, uint32_t group, uint32_t words_per_query, uint32_t max_block_words, uint32_t waves)
	Splits a tile into refine_cells, about waves cells per block when the tile allows, so that blocks claiming cells dynamically even out cells of uneven work.
template<bool UpperTriangle>
__host__ __device__ uint64_t	batch_result_slot (uint32_t query_index, uint32_t reference_id, uint32_t query_id_offset, uint32_t reference_count) noexcept
	Position of the pair (query_index of the tile, reference_id) in a tile's results: rows of reference_count references per query, or for all-to-all tiles the packed strict upper triangle of pairs with query_id < reference_id, where query IDs start at query_id_offset.
template<size_t BucketCount, bool CountMatches, typename SearchResult , refine_candidates Candidates, bool UpperTriangle, uint32_t QueryGroup>
__global__	__launch_bounds__ (bitmap_refine_block_size) void refine_batch_bitmap_kernel(uint32_t const *query_planes
	Exactly compares query bit-planes with references, sharing each reference row across a query group.
__device__ __forceinline__ void	load_plane_chunk (uint4 const *row, uint32_t group, uint32_t chunk, uint32_t(&words)[4])
	Loads plane chunk chunk (planes 4 chunk ... 4 chunk + 3) of group group of a bit-plane row (score_plane_index layout).
template<size_t BucketCount, bool UpperTriangle, uint32_t QueryGroup>
__global__ void	match_filter_kernel (uint32_t const *query_planes, uint32_t query_id_offset, uint32_t query_count, uint32_t const *reference_planes, uint32_t reference_count, uint32_t indexed_bucket_count, uint32_t minimum_matches, uint32_t *candidate_bits, uint32_t *cell_counter)
	Records in candidate_bits (candidate-bitmap layout) the pairs of the layout whose scores agree in their low match_filter_planes planes in at least minimum_matches buckets.
template<size_t BucketCount, typename SearchResult , bool UpperTriangle, uint32_t QueryGroup>
__global__ void	refine_all_pairs_kernel (uint32_t const *query_planes, uint32_t query_id_offset, uint32_t query_count, uint32_t const *reference_planes, uint32_t reference_count, SearchResult *results, uint32_t *cell_counter, uint32_t const *reference_mask)
	Exactly compares the layout pairs selected by reference_mask, without match counts.
template<size_t BucketCount, typename SearchResult , bool UpperTriangle, bool Swar>
__global__	__launch_bounds__ (block_size) void row_major_all_pairs_kernel(uint16_t const *rows
	The all-pairs comparison of refine_all_pairs_kernel over row-major 16-bit scores instead of bit-planes, one warp per pair, for ablation builds to compare with.
template<size_t BucketCount, typename Layout = default_register_layout>
__global__ void	cardinality_kernel (uint32_t const *registers, uint64_t *empty_out, double *estimate_out)
	Computes the cardinality of a single constructed sketch.
template<size_t BucketCount, typename Layout = default_register_layout>
__global__ void	hybrid_cardinality_kernel (uint32_t const *const registers, hybrid_cardinality_estimates *const estimates)
	Computes BBTools and paper-style HybridDDL estimates in one register scan.
template<size_t BucketCount, hybrid_variant Variant, typename Layout = default_register_layout>
__global__ void	hybrid_cardinality_variant_kernel (uint32_t const *const registers, double *const estimate)
	Computes one HybridDDL variant estimate for estimator comparisons.
template<size_t BucketCount, typename Layout = default_register_layout>
__global__ void	batch_cardinality_kernel (uint32_t const *const registers, uint32_t const row_count, uint64_t *const empty_out, double *const estimates_out)
	Computes the cardinality of every stored sketch in one launch.
template<size_t BucketCount>
__global__ void	batch_scores_kernel (uint32_t const *const registers, uint32_t const row_count, uint16_t *const scores_out)
	Copies the winning score of every register into compact row-major scores.
cuda::pinned_memory_pool_ref	device_pinned_pool (cuda::device_ref device, pinned_placement placement)
	A page-locked memory pool accessible from device only.
template<typename Layout = default_register_layout>
__host__ __device__ uint16_t	score (uint64_t hash) noexcept
	Computes the 16-bit DDL rarity score for hash.
template<typename Layout = default_register_layout>
__host__ __device__ constexpr uint64_t	restore (uint16_t stored) noexcept
	Reconstructs the approximate original hash magnitude from a stored score.
template<typename Layout = default_register_layout>
__host__ __device__ constexpr uint64_t	restore_midpoint (uint16_t stored) noexcept
	Reconstructs the interval midpoint for restore.
__device__ __forceinline__ uint32_t	load_shared_relaxed (uint32_t const *address)
	Relaxed CTA-scope load of a shared word that other threads update atomically.
template<size_t BucketCount, typename Layout , bool HasBlacklist = false, bool Naive = false>
__device__ __forceinline__ void	add_sequence_windows (char const *sequence, uint32_t windows, size_t first_tile, size_t tile_stride, uint32_t k, uint32_t *registers, bool packed_input=false, blacklist_view blacklist={})
	Encodes raw ASCII windows into registers using shared overlapping tiles.
template<size_t BucketCount, typename Layout , bool HasBlacklist = false, bool Naive = false>
__global__ void	add_sequence_batch_kernel (sequence_batch_chunk const *chunks, size_t chunk_count, size_t block_count, uint32_t k, uint32_t *registers, blacklist_view blacklist={})
	Batch kernel covering every staged chunk in one launch, including short records.
template<size_t BucketCount, typename Layout , bool HasBlacklist = false>
__global__ void	add_sequence_single_kernel (char const *sequence, uint32_t windows, uint32_t k, uint32_t *registers, blacklist_view blacklist={})
	Single-sequence kernel for the public raw-ASCII API.
template<size_t BucketCount, typename Layout = default_register_layout>
__host__ Result< void >	launch_sequence_add (device_span< char const > sequence, uint32_t k, device_span< uint32_t > registers, cuda::stream_ref stream, cuda::std::optional< blacklist_view > blacklist=cuda::std::nullopt)
	Shared launch for one device-resident ASCII chunk.
Result< void >	mpi_try (int status, char const *call)
Result< void >	agree (Result< void > local, MPI_Comm comm)
	Returns local on the rank that failed, and on every other rank an error naming the lowest failing rank, so that all ranks leave a collective step together.
Result< std::vector< std::filesystem::path > >	share_listing (std::filesystem::path const &input, int rank, MPI_Comm comm)
	Rank 0's listing of input on every rank.
size_t	slice_begin (std::span< uint64_t const > bytes, uint64_t total, int rank, int size)
	First genome of rank rank's slice: contiguous slices of about equal bytes.
Result< std::vector< uint32_t > >	slice_counts (reference_database_file const &mine, genome_slice const &slice, MPI_Comm comm)
	Every rank's genome count, after checking collectively that mine holds the sketches of this rank's slice and that every rank built with rank 0's settings.
Result< MPI_Datatype >	row_type (uint32_t buckets)
	One row of buckets scores as a single MPI element, so counts of rows stay within int.
template<typename Search >
Result< void >	shared_search (MPI_Comm comm, Search &&search)
	Runs search with a replica_search that shares tiles across the ranks of comm.
template<typename T >
T	try_unwrap_success (Result< T > &result)
void	try_unwrap_success (Result< void > &result)
cuda::std::unexpected< Error >	propagate_error (const Error &error)
	Copies error for propagation (avoids moving out of expected::error()).
__host__ __device__ constexpr uint32_t	load_256_alignment () noexcept
	Byte alignment load_256_global_nc requires on the compiled architecture.
template<typename QueryScore , typename ReferenceScore >
__host__ __device__ bool	wide_rows_aligned (QueryScore const *query, ReferenceScore const *reference) noexcept
	True when both row bases meet load_256_alignment on the compiled architecture.
template<typename T >
__device__ __forceinline__ void	load_256_global_nc (const T *ptr, T *out)
	Loads 256 bits from global memory using the non-coherent cache path.
template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout>
Result< void >	validate_score_compatibility (score_compatibility const &compatibility)
template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout>
Result< void >	validate_non_indexed_score_compatibility (score_compatibility const &compatibility)
template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout>
Result< void >	validate_indexed_score_compatibility (score_compatibility const &compatibility)
constexpr uint64_t	indexed_cell_count (score_compatibility const &compatibility) noexcept
constexpr uint64_t	indexed_posting_count (uint32_t reference_count, score_compatibility const &compatibility) noexcept
constexpr index_storage	smaller_index_storage (uint32_t reference_count, score_compatibility const &compatibility) noexcept
	Storage needing fewer device bytes for reference_count references; dense on ties.
uintptr_t	align_up (uintptr_t address, size_t alignment) noexcept
constexpr uint32_t	batch_query_tile_size (uint32_t reference_count, uint32_t query_count, uint32_t tile_limit=batch_query_tile_count) noexcept
constexpr uint32_t	all_to_all_tile_queries (uint32_t reference_count, uint32_t parts=1U) noexcept
	Query rows one all-to-all tile compares: about 2^25 pairs, and at least batch_query_tile_count.
uint32_t	external_batch_query_limit (uint32_t reference_count, cuda::device_ref device)
	Query rows one external batch tile may hold on device.
template<typename T >
T	database_file_little_endian (T value)
template<typename IO >
Result< void >	database_file_metadata (IO &io, reference_database_metadata &metadata, uint32_t &blacklist_count)
Result< uint64_t >	reference_database_digest (reference_database_file const &db)
__host__ __device__ constexpr uint64_t	top_k_key (pairwise_counts const &counts, uint32_t id) noexcept
	Ranking key of a neighbour: its weighted k-mer identity above its complemented ID.
constexpr size_t	top_k_heap_words (uint32_t k) noexcept
	Words of shared memory one heap takes: k keys, then their k packed counts.
__device__ void	top_k_replace_root (uint64_t *heap, uint32_t k, uint64_t key, uint64_t value)
	Replaces the root of a min-heap of k keys at heap, whose values follow the keys, with key and value, and sifts it down.
template<bool Thresholded, bool UpperTriangle>
__device__ uint32_t	top_k_word_bits (batch_result_tile const &tile, uint32_t query, uint32_t word, uint32_t words)
	The pass bits of word word of tile row query: the stored bits, or for exhaustive tiles every reference of the word.
template<bool UpperTriangle>
__device__ packed_pairwise_counts	top_k_stored (batch_result_tile const &tile, uint32_t query, uint32_t reference)
	The stored result of tile row query and reference, which must pass.
__device__ uint32_t	top_k_nth_bit (uint32_t mask, uint32_t n)
	Position of set bit n, counting from zero, of mask, which has more than n set bits.
template<typename Bits , typename Entry >
__device__ void	top_k_warp_merge (uint64_t *heap, uint32_t k, uint32_t words, Bits bits_of, Entry entry_of)
	One warp merges the candidates of words bitmap words into the heap of k entries at heap, in shared memory, which only this warp updates, so it needs no lock.
template<bool ToShared>
__device__ void	top_k_warp_copy (top_k_slots const &slots, size_t row, uint64_t *heap)
	Copies the k entries of global heap row into, or for ToShared false out of, the shared heap heap with the warp's lanes.
	if (query >=tile.query_count) return
	top_k_warp_copy< true > (slots, row, heap)
	top_k_warp_merge (heap, slots.k, words - cuda::std::min(skipped, words), [&](uint32_t word) { return top_k_word_bits< Thresholded, UpperTriangle >(tile, query, skipped+word, words);}, [&](uint32_t candidate, uint64_t &key, uint64_t &value) { auto const reference=skipped *32U+candidate;auto const stored=top_k_stored< UpperTriangle >(tile, query, reference);key=top_k_key(stored.unpack(tile.bucket_count), reference);value=stored.bits;})
	top_k_warp_copy< false > (slots, row, heap)
	if (block >=static_cast< uint64_t >(row_words) *word_count) return
	_Pragma ("unroll") for(uint32_t bit=0U

Variables
constexpr std::array< std::string_view, 22 >	ablations
	Optimisations that the CUDDL_ABLATE environment variable can turn off in a build configured with -Dablations=true (CUDDL_ABLATIONS=1).
constexpr bool	ablations_enabled = false
constexpr double	hash_range = 0x1p64
	2^64, the range of the 64-bit hash used by the sketch.
constexpr size_t	default_arena_ceiling = size_t{128} << 20
	Default arena ceiling.
constexpr size_t	stager_arenas = 4
	Staging arenas the copy stream rotates through, so copies run ahead of the kernels that read earlier arenas while other work shares the device.
constexpr size_t	stager_hold_slots = 16
	Host memory the device still has to read must stay alive: the transfer engine's read of a pageable buffer is not ordered with the host writes that follow it.
constexpr size_t	packed_buffer_bytes = size_t{2} << 20
	Bytes reserved per loader buffer that holds packed bases: a bacterial genome's two-bit words, about 1.5 MB, fit.
constexpr uint32_t	missing_cell = 0x00FFFFFFU
	Eight ambiguous bases: an encoder cell past the end of a sequence.
constexpr size_t	parallel_read_depth = 64
	Reads a parallel file system needs in flight to give one node its bandwidth: on MOGON's Lustre, 16 threads read 2.9 GB/s of ~1 MB files and 64 threads 6.7 GB/s.
constexpr uint64_t	seed = 42
	Domain separation constant folded into every packed k-mer before hashing.
constexpr uint64_t	PRIME64_1 = 11400714785074694791ULL
	xxHash-64 round primes.
constexpr uint64_t	PRIME64_2 = 14029467366897019727ULL
constexpr uint64_t	PRIME64_3 = 1609587929392839161ULL
constexpr uint64_t	PRIME64_4 = 9650029242287828579ULL
constexpr uint64_t	PRIME64_5 = 2870177450012600261ULL
constexpr int32_t	nlz_bins = 66
	NLZ histogram bins for the hybrid estimator, including the empty tier.
constexpr double	dlc_blend_low = 0.088
constexpr double	dlc_blend_high = 0.952
constexpr double	dlc_min_fraction = 0.002
constexpr double	dlc_info_power = 4.5
constexpr double	hybrid_low = 0.2
constexpr double	hybrid_high = 5.0
constexpr uint32_t	ddl_cf_size = 13
constexpr uint32_t	block_size = 256
	Threads per CTA for the single-pair and cardinality reduction kernels.
constexpr size_t	maximum_batch_rows = (size_t{1} << 31) - 1U
	Stored sketches one batch launch may index, bounded by the CUDA grid dimension.
template<typename QueryScore , typename ReferenceScore >
constexpr uint32_t	wide_chunk_buckets
	Buckets one contiguous chunk covers, for a query/reference score pair.
constexpr uint32_t	shared_construction_block_size = 768
	Threads per CTA for the CTA-local construction kernel.
__global__ size_t	input_size
__global__ size_t uint32_t *	registers
__global__ size_t uint32_t bool	vector_input
__global__ size_t uint32_t bool blacklist_view	blacklist
__global__ size_t uint32_t bool blacklist_view bool	IncludeCardinality
__global__ size_t uint32_t bool blacklist_view bool	Layout
constexpr uint32_t	index_match_cells_per_warp = 32U
	Cells one warp owns per iteration of count_batch_index_matches_kernel: one per lane.
__global__ uint32_t	reference_count
__global__ uint32_t uint32_t	indexed_bucket_count
__global__ uint32_t uint32_t uint32_t *	directory
auto const	shift = sparse_directory_shift(reference_count)
auto const	total = static_cast<size_t>(indexed_bucket_count) * entries
__global__ uint16_t const *	sorted_keys
__global__ uint16_t const uint32_t uint32_t uint32_t	key_count = static_cast<uint32_t>(key_mask) + 1U
__global__ uint16_t const uint32_t uint32_t uint32_t unsigned long long *	work
unsigned long long	local = 0U
constexpr uint32_t	posting_range_block_size = 1024U
	Threads of the single index_posting_ranges_kernel block.
static __global__ uint32_t const *	offsets
static __global__ uint32_t const uint16_t const uint32_t uint32_t uint16_t	key_mask
static __global__ uint32_t const uint16_t const uint32_t uint32_t uint16_t uint32_t *	range_begin
static __global__ uint32_t const uint16_t const uint32_t uint32_t uint16_t uint32_t uint32_t *	range_end_scan
__shared__ block_scan::TempStorage	storage
uint32_t	carry = 0U
static __global__ uint32_t const uint32_t uint32_t const *	postings
static __global__ uint32_t const uint32_t uint32_t const uint32_t *	match_counts
auto const	stride = static_cast<uint32_t>(gridDim.x) * block_size
__global__ size_t	query_row_offset
__global__ size_t uint32_t	query_count
constexpr uint32_t	index_tile_lane_postings = 16U
	Longest posting range one count_batch_index_tile_kernel lane walks alone.
constexpr uint32_t	index_tile_long_capacity = 32U
	Long posting ranges one count_batch_index_tile_kernel block walks together.
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t	tile
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const *	ranges
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t	minimum_matches
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t	query_id_offset
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool	all_to_all
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool uint32_t *	candidate_bits
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool uint32_t uint32_t const *	bitmaps
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool uint32_t uint32_t const uint32_t const *	bitmap_starts
__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool uint32_t uint32_t const uint32_t const uint32_t const *	bitmap_counts
constexpr uint32_t	warps_per_block = index_tile_block_size / warp_width
__shared__ uint32_t	packed_counts []
auto const	tiles = (reference_count + tile - 1U) / tile
auto const	query_index = static_cast<uint32_t>(blockIdx.x) / tiles
auto	low = static_cast<uint32_t>(blockIdx.x) % tiles * tile
auto const	high = min(low + tile, reference_count)
auto const	width = high - low
__shared__ uint2	long_ranges [index_tile_long_capacity]
__shared__ uint32_t	long_count
auto const	lane = static_cast<uint32_t>(threadIdx.x) % warp_width
auto const	warp = static_cast<uint32_t>(threadIdx.x) / warp_width
	else
auto const *	query = queries + (query_row_offset + query_index) * BucketCount
auto const	count
auto const	long_ranges_used
auto *const	bits
constexpr size_t	refine_block_row_bytes = size_t{8} << 20U
	Reference-row bytes one refine_batch_index_candidates_kernel block spans.
constexpr uint32_t	score_planes = 16U
	Bits per score, and so bit-planes per 32-bucket group.
__global__ uint32_t	row_count
__global__ uint32_t uint32_t *	planes
constexpr uint32_t	groups_per_row = static_cast<uint32_t>(BucketCount / warp_width)
__global__ size_t uint16_t *	rows
__global__ uint32_t const *	reference_planes
__global__ uint32_t const uint32_t uint32_t const *	candidate_ids
__global__ uint32_t const uint32_t uint32_t const uint32_t const *	candidate_count
__global__ uint32_t const uint32_t uint32_t const uint32_t const SearchResult *	results
constexpr uint32_t	row_words = static_cast<uint32_t>(BucketCount / 2U)
auto const	items = candidate_ids == nullptr ? reference_count : *candidate_count
auto const	pairs = static_cast<uint64_t>(query_count) * reference_count
auto const	indexed_groups = indexed_bucket_count / warp_width
constexpr uint32_t	bitmap_refine_block_size = 768U
	Threads per refine_batch_bitmap_kernel block.
constexpr uint32_t	bitmap_refine_waves = 8U
	Cells per refine_batch_bitmap_kernel block that a tile of bitmap-selected pairs is split into when it allows.
constexpr size_t	bitmap_refine_shared_bytes = 96U * 1024U
	Shared bytes refine_batch_bitmap_kernel may stage query planes in; one block per SM fits on every supported architecture.
template<size_t BucketCount>
constexpr bool	bitmap_refine_staged = bitmap_refine_shared_bytes >= BucketCount * sizeof(uint16_t)
	Whether a query group's planes fit bitmap_refine_shared_bytes.
template<size_t BucketCount>
constexpr uint32_t	bitmap_refine_group
	Queries one refine_batch_bitmap_kernel block refines together.
template<size_t BucketCount>
constexpr size_t	bitmap_refine_dynamic_bytes
	Shared bytes one refine_batch_bitmap_kernel block stages.
__global__ uint32_t const uint32_t const *	pass_bits
__global__ uint32_t const uint32_t const uint32_t const *	word_offsets
__global__ uint32_t const uint32_t const uint32_t const uint32_t	first_query_id
__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t	word_begin
__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t	word_end
__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t uint32_t	packed_base
__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t uint32_t SearchResult *	packed
__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t uint32_t SearchResult uint32_t *	packed_match_counts
auto const	words_per_query = candidate_bit_words(reference_count)
__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t uint32_t const SearchResult uint32_t *	result_match_counts
__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t uint32_t const SearchResult uint32_t uint32_t uint32_t uint32_t *	cell_counter
constexpr uint32_t	group = QueryGroup
constexpr bool	staged = bitmap_refine_staged<BucketCount>
constexpr uint32_t	lane_groups = groups_per_row / warp_width
constexpr uint32_t	cached_groups = lane_groups < 2U ? lane_groups : 2U
constexpr uint32_t	max_block_words
constexpr uint32_t	word_slices = Candidates == refine_candidates::all ? 1U : 4U
__shared__ uint4	query_plane_storage []
__shared__ uint32_t	next_word
__shared__ uint32_t	claimed_cell
auto const	cells
auto const	groups = cells.groups
auto const	thresholded = CountMatches && pass_bits != nullptr
constexpr uint32_t	match_filter_block_size = 512U
	Threads per match_filter_kernel block.
template<size_t BucketCount>
constexpr bool	match_filter_supported
	Whether match_filter_kernel supports BucketCount: a lane caches at most two plane groups of each of its references.
constexpr uint32_t	match_filter_planes = 12U
	Low planes match_filter_kernel compares.
constexpr uint32_t	all_pairs_block_size = 512U
	Threads per refine_all_pairs_kernel block.
constexpr uint32_t	all_pairs_word_parts = 4U
	Parts a refine_all_pairs_kernel warp claims each 32-reference word in.
__global__ uint32_t uint32_t uint32_t SearchResult uint32_t const *	reference_mask
auto const	warps = uint64_t{gridDim.x} * (block_size / warp_width)
constexpr uint32_t	delta_change_limit = 160U
	Most buckets in which a child reference may differ from its base reference.
constexpr uint32_t	delta_signatures = 8U
	Exact bucket signatures that propose base candidates, and buckets per signature.
constexpr uint32_t	delta_signature_buckets = 8U
constexpr uint32_t	delta_signature_candidates = 4U
	Highest-ID rows taken from each matching signature.
constexpr uint32_t	delta_candidates = delta_signatures * delta_signature_candidates
constexpr uint32_t	delta_correction_warps = 24U
	Warps per apply_reference_deltas_kernel block, and the most and fewest children one block cell takes.
constexpr uint32_t	delta_child_tile = 2048U
constexpr uint32_t	delta_min_child_tile = 4U * delta_correction_warps
template<size_t BucketCount>
constexpr uint32_t	delta_query_group = static_cast<uint32_t>(32768U / BucketCount)
	Queries one correction cell stages bucket-major: 64 KiB of scores for every bucket count.
template<size_t BucketCount>
constexpr size_t	delta_correction_shared_bytes
__global__ uint32_t uint32_t	signature
__global__ uint32_t uint32_t uint64_t *	keys
__global__ uint32_t uint32_t uint64_t uint32_t *	ids
constexpr uint8_t	mantissa_bits = default_register_layout::mantissa_bits
	Default score layout retained for source compatibility.
constexpr uint32_t	mantissa_mask = (1U << mantissa_bits) - 1U
constexpr uint32_t	restore_shift_base = 63U - mantissa_bits
	Default restoration constants retained for source compatibility.
constexpr uint32_t	max_nlz = restore_shift_base
constexpr double	index_pair_fraction_limit = 0.03
	Index pair fraction (reference_index::pair_fraction) above which batch search compares bit-planes for every pair instead of counting through the index.
constexpr uint32_t	batch_query_tile_count = 128U
	Minimum query rows per batch tile.
constexpr uint32_t	top_k_warps = block_size / 32U
	Warps per merging block, each holding one heap; fewer when the heaps exceed shared memory.
__global__ top_k_slots	slots
auto *const	heap = heaps + warp * top_k_heap_words(slots.k)
auto const	query_id = tile.first_query_id + query
auto const	row = static_cast<size_t>(query_id - slots.first_genome_id)
auto const	words = tile.words_per_query()
auto const	skipped = UpperTriangle ? (query_id + 1U) / 32U : 0U
__global__ uint32_t *	columns
__global__ uint32_t uint32_t	first_word
auto const	block = static_cast<uint64_t>(blockIdx.x) * (blockDim.x / 32U) + threadIdx.x / 32U
auto const	word_count = words - first_word
auto const	word = first_word + static_cast<uint32_t>(block % word_count)
auto const	row_word = static_cast<uint32_t>(block / word_count)
uint32_t	column = 0U

Typedef Documentation

device_scope

using cuddl::detail::device_scope = typedef ::cuda::__ensure_current_context

Makes stream's device current for the enclosing scope and restores the previous context on exit.

Every public function that takes a stream opens one, so its kernel launches, CUB calls and function attributes target the stream's device, never the calling thread's current device. CCCL provides this only as an internal type.

Definition at line 55 of file reference_database.cuh.

fastx_line_end_fn

using cuddl::detail::fastx_line_end_fn = typedef decltype(&fastx_line_end_scalar)

Definition at line 175 of file fastx_sequence_file.hpp.

Enumeration Type Documentation

decompression_backend

enum class cuddl::detail::decompression_backend : uint8_t

strong

Where inputs are decompressed; sketch construction always runs on the GPU.

Enumerator
automatic
hybrid	CPU inflation, followed by GPU FASTA normalisation of the inflated bytes.
cpu	CPU inflation and normalisation.
gpu	nvCOMP for eligible gzip inputs; unsupported formats use host loaders.
gpu_cores	As gpu, with nvCOMP inflating on the SMs even where the device has a decompression engine: the baseline that shows what the engine adds.

Definition at line 46 of file database_staging.cuh.

                                 : uint8_t {
    automatic,
    hybrid,
    cpu,
    gpu,
    gpu_cores,
};

fastx_format

enum class cuddl::detail::fastx_format : uint8_t

strong

Detected record format for a FASTX buffer.

Enumerator
unknown	Format not yet determined from the first header.
fasta	FASTA (> headers).
fastq	FASTQ (@ headers).

Definition at line 438 of file fastx_sequence_file.hpp.

                        : uint8_t {
    unknown,
    fasta,
    fastq,
};

hybrid_variant

enum class cuddl::detail::hybrid_variant : uint8_t

strong

Which HybridDDL blend the variant kernel computes.

Enumerator
bbtools
paper

Definition at line 21 of file detail/hybrid_cardinality.cuh.

                          : uint8_t {
    bbtools,
    paper,
};

pinned_placement

enum class cuddl::detail::pinned_placement : uint8_t

strong

Where a device_pinned_pool takes its pages.

Enumerator
device_node	The device's host NUMA node, where the system supports NUMA pools: for buffers the device writes, such as downloaded results.
host	Any node: for buffers host threads fill. Sketch loaders run on every node, and on an A40 node with 16 NUMA nodes and two GPUs per node, device-node loader buffers sketched 100,000 genomes in 18.6 s against 15.1 s.

Definition at line 19 of file pinned_pool.cuh.

                            : uint8_t {
    device_node,
    host,
};

refine_candidates

enum class cuddl::detail::refine_candidates

strong

Pairs refine_batch_bitmap_kernel compares.

Enumerator
bitmap	Exactly the pairs set in a pass bitmap.
all	Every pair of the layout, or with pass bits requested, those meeting the threshold.
reference_mask	Every pair of the layout whose reference is set in one reference-wide bitmap.

Definition at line 1737 of file kernels.cuh.

                             {
    bitmap,
    all,
    reference_mask,
};

transfer_mode

enum class cuddl::detail::transfer_mode : uint8_t

strong

How a build gets CPU-parsed sequence bytes to the device.

Enumerator
automatic	In-place host reads on coherent devices, page-locked transfers elsewhere.
in_place	Decompress into a heap buffer and let the kernels read it there. Only a device that reads pageable host memory can do this, and it pays neither the copy nor the page-locking.
pinned	Decompress into page-locked buffers, which the transfer engine reads directly. Measured 2.1x faster than staging on an x86 host with a discrete GPU, and slower on a coherent system, where the same mapping costs host writes more than the copy it removes.
staged	Decompress into a heap buffer and copy it to the device. On a coherent system that copy is a single-threaded bounce through the driver, which is what leaves 72 loaders parked while one core copies and the device waits for data.

Definition at line 60 of file database_staging.cuh.

                         : uint8_t {
    automatic,
    in_place,
    pinned,
    staged,
};

Function Documentation

__launch_bounds__() [1/6]

template<size_t BucketCount, bool CountMatches, typename SearchResult , refine_candidates Candidates, bool UpperTriangle, uint32_t QueryGroup>

__global__ cuddl::detail::__launch_bounds__

(

bitmap_refine_block_size

)

const

Exactly compares query bit-planes with references, sharing each reference row across a query group.

Block cells are (reference block, group of bitmap_refine_group queries), reference-block major so resident blocks share reference rows through L2. A block stages its group's query bit-planes (score_plane_index) in shared memory when they fit; warps then claim 32-reference words and, for every reference any group query selects, compare the reference's bit-planes with each selecting query. A row thus crosses L2 once per group rather than once per pair.

Each written result lands at its batch_result_slot. Candidates selects which pairs are written: those in candidate_bits (one word per query and 32 references), or every pair of the layout. With pass_bits (counting kernels only), only the selected pairs with at least minimum_matches index matches are written and recorded in pass_bits, in the candidate-bitmap layout. In bitmap mode pass_bits must be candidate_bits, whose failing pairs' bits are cleared in place. With result_match_counts, results also get their index match counts, recomputed from the planes.

__launch_bounds__() [2/6]

__global__ cuddl::detail::__launch_bounds__

(

block_size

)

const

Builds a sparse key directory: entry c of bucket b is the first position in b's sorted keys whose key is at least c << shift.

Marks the first position of every run of equal sorted keys with its index.

Turns per-pair match counts into the query-major candidate bitmap.

Counts the query's posting matches for every reference, one posting per thread.

Adds the squared length of every nonzero-key posting list to work.

Dense indexes pass their cell offsets (key_count cells per bucket); sparse indexes pass their per-bucket sorted keys instead, whose runs are the posting lists. The sum is the number of (reference, reference, bucket) triples sharing an index key: an index walks about work / reference_count postings per database-like query.

__launch_bounds__() [3/6]

template<size_t BucketCount>

__global__ cuddl::detail::__launch_bounds__

(

block_size

)

const

Transposes compact score rows into the bit-plane layout of score_plane_index.

Transposes an all-to-all tile's pass bits from first_word on into per-genome columns: word w * reference_count + g of columns holds which of tile rows 32 w to 32 w + 31 pass with genome g.

Hashes signature signature of every row; rows are emitted in descending ID order so a stable sort keeps the highest IDs first within a signature.

Counts every query/reference pair's index matches from bit-planes, one warp per pair.

Recovers row-major scores from bit-plane rows, one warp per 32-bucket group.

Used when a search asks for index semantics without an index: the counts equal the ones the index postings would produce.

One warp per 32-by-32 bit block, which its lanes load a row each and turn with 32 ballots; neighbouring warps take neighbouring words of the same rows, so they share the rows' sectors, and every warp writes 32 adjacent words.

__launch_bounds__() [4/6]

template<size_t BucketCount, typename QueryRow >

__global__ cuddl::detail::__launch_bounds__

(

block_size

)

const

Counts index matches for every query/reference pair in one tile.

One warp per tile row merges the row's passing pairs into the row genome's heap.

Packs a threshold tile's passing results in order.

Compares one query's bit-planes with every reference, or with the selected candidates.

Resolves every query's sparse posting range for one indexed bucket per block.

Each lane of a warp resolves the posting range of its own (query, bucket) cell, so a warp keeps 32 independent range lookups in flight. The warp then walks every non-empty range with a lane stride, so hot keys with long lists (the dominant cost on skewed rows) are consumed 32 postings at a time.

Block b owns bucket b, so its lookups share the bucket's directory and keys through L1. ranges is query-major, one entry per (query, indexed bucket).

One warp compares one reference; lane l takes groups l, l + 32, ... With candidate_ids, item i refines reference candidate_ids[i] for *candidate_count items; otherwise item i is reference i.

One warp per pass-bit word in [word_begin, word_end); lane l handles reference 32 * word + l, so consecutive lanes read consecutive slots and write consecutive packed positions. word_offsets is the exclusive scan of the word popcounts; results land at their packed position minus packed_base.

__launch_bounds__() [5/6]

template<size_t BucketCount, typename SearchResult , bool UpperTriangle, bool Swar>

__global__ cuddl::detail::__launch_bounds__

(

block_size

)

const

The all-pairs comparison of refine_all_pairs_kernel over row-major 16-bit scores instead of bit-planes, one warp per pair, for ablation builds to compare with.

With Swar each lane compares 16 buckets per step from 256-bit loads, two packed scores per 32-bit subtraction (classify_query_words); without, it classifies one bucket at a time. rows holds every reference's row, and query q of the tile is row query_id_offset + q.

__launch_bounds__() [6/6]

template<size_t BucketCount, typename Layout = default_register_layout, bool HasBlacklist = false>

__global__ cuddl::detail::__launch_bounds__

(

shared_construction_block_size

)

const

Constructs a sketch through a CTA-local shared-memory register array.

Each item costs a hash, a score, and one fire-and-forget shared atomic max; the CTA then merges its registers into the global sketch with one global atomic max per nonempty bucket.

The host launches two CTAs per SM and the kernel walks the input with a runtime grid-stride loop over 256-bit chunks, so the grid is a single balanced wave for every input size and the per-CTA merge traffic stays minimal.

__maxnreg__()

template<size_t BucketCount, uint32_t CounterBits, typename QueryRow >

__global__ cuddl::detail::__maxnreg__

(

48

)

const

Counts index matches for one query against one reference tile in shared memory.

Block b owns query b / tiles and references [t * tile, (t + 1) * tile) for t = b % tiles, with two 16-bit counters per shared word. A query's tiles run as neighbouring blocks, so they share its posting-range lookups through the caches. Postings ascend by reference within each key, so each posting list is narrowed to the tile with two binary searches. The flush writes every pair of the tile, so match_counts needs no zeroing. With nonzero CounterBits, each lane first sums the query's posting bitmaps over one word of the tile in that many bit-sliced counters, seeding references that already reach minimum_matches with exactly that count; the posting lists then add the rest.

Launched with index_tile_block_size threads: the shared counters, not the thread count, bound how many blocks fit on an SM, so larger blocks keep more warps resident to hide the posting loads' latency. Capping registers at 48 leaves a quarter of the SM's registers to kernels of neighbouring tiles; a GH200 searched 115,223 genomes 12% faster than with 60.

__syncthreads()

cuddl::detail::__syncthreads

(

)

Here is the caller graph for this function:

_Pragma()

cuddl::detail::_Pragma ( "unroll"  )

pure virtual

Here is the caller graph for this function:

ablated()

constexpr bool cuddl::detail::ablated ( std::string_view  )

constexprnoexcept

Always false: this build compiles the ablation switches out.

Definition at line 82 of file ablation.hpp.

                                                {
    return false;
}

Here is the caller graph for this function:

acquire_sequence_target()

decompression_target cuddl::detail::acquire_sequence_target ( void *  context, size_t  bytes  )

inline

decompression_source trampoline for sequence_buffer_pool.

Definition at line 1012 of file database_staging.cuh.

                                                                                               {
    return static_cast<sequence_buffer_pool*>(context)->acquire(bytes);
}

Here is the caller graph for this function:

add_sequence_batch_kernel()

template<size_t BucketCount, typename Layout , bool HasBlacklist = false, bool Naive = false>

__global__ void cuddl::detail::add_sequence_batch_kernel	(	sequence_batch_chunk const *	chunks,
		size_t	chunk_count,
		size_t	block_count,
		uint32_t	k,
		uint32_t *	registers,
		blacklist_view	blacklist = {}
	)

Batch kernel covering every staged chunk in one launch, including short records.

The register cap keeps the blacklist lookup from costing resident blocks: at 43 registers, a GH200 SM held five blocks instead of eight.

Definition at line 304 of file sequence_encode.cuh.

                               {}
) {
    __shared__ sequence_batch_chunk chunk;
    __shared__ size_t first_block;
    for (size_t block = blockIdx.x; block < block_count; block += gridDim.x) {
        if (threadIdx.x == 0) {
            size_t low = 0, high = chunk_count;
            while (low < high) {
                auto const middle = low + (high - low) / 2;
                if (chunks[middle].block_end <= block) {
                    low = middle + 1;
                } else {
                    high = middle;
                }
            }
            chunk = chunks[low];
            first_block = low ? chunks[low - 1].block_end : 0;
        }
        __syncthreads();
        auto* target = registers + size_t{chunk.genome} * BucketCount;
        add_sequence_windows<BucketCount, Layout, HasBlacklist, Naive>(
            chunk.bases,
            chunk.windows,
            (block - first_block) * 2048,
            (chunk.block_end - first_block) * 2048,
            k,
            target,
            chunk.packed != 0,
            blacklist
        );
        __syncthreads();
    }
}

add_sequence_single_kernel()

template<size_t BucketCount, typename Layout , bool HasBlacklist = false>

__global__ void cuddl::detail::add_sequence_single_kernel	(	char const *	sequence,
		uint32_t	windows,
		uint32_t	k,
		uint32_t *	registers,
		blacklist_view	blacklist = {}
	)

Single-sequence kernel for the public raw-ASCII API.

There is no cross-chunk carry: the caller supplies any K-1 overlap explicitly.

Definition at line 350 of file sequence_encode.cuh.

                               {}
) {
    add_sequence_windows<BucketCount, Layout, HasBlacklist>(
        sequence,
        windows,
        size_t{blockIdx.x} * 2048,
        size_t{gridDim.x} * 2048,
        k,
        registers,
        false,
        blacklist
    );
}

add_sequence_windows()

template<size_t BucketCount, typename Layout , bool HasBlacklist = false, bool Naive = false>

__device__ __forceinline__ void cuddl::detail::add_sequence_windows	(	char const *	sequence,
		uint32_t	windows,
		size_t	first_tile,
		size_t	tile_stride,
		uint32_t	k,
		uint32_t *	registers,
		bool	packed_input = false,
		blacklist_view	blacklist = {}
	)

Encodes raw ASCII windows into registers using shared overlapping tiles.

Each shared word holds eight two-bit bases and eight ambiguity bits. Threads reuse neighboring words to construct eight overlapping windows on both strands. Naive instead builds every window base by base with a branch per base, for ablation studies only.

Definition at line 44 of file sequence_encode.cuh.

                               {}
) {
    constexpr uint32_t tile_size = 256 * 8;
    __shared__ uint32_t cells[tile_size / 8 + 4];
    __shared__ uint32_t local[BucketCount];
    for (uint32_t i = threadIdx.x; i < BucketCount; i += blockDim.x) {
        local[i] = 0;
    }
    uint32_t hash_ceiling = UINT32_MAX;
    size_t tiles = 0;
    auto const mask = (uint64_t{1} << (2 * k)) - 1;
    auto const valid_mask = (uint32_t{1} << k) - 1;
    for (size_t tile = first_tile; tile < windows; tile += tile_stride) {
        auto const count = cuda::std::min(tile_size, windows - static_cast<uint32_t>(tile));
        if constexpr (Naive) {
            // The tiled path's cell fill ends with the barrier that orders the zeroing of
            // @c local before its first update; this path has no fill, so it waits here.
            __syncthreads();
            auto const start = threadIdx.x * 8;
            auto const* const words = reinterpret_cast<uint32_t const*>(sequence);
            for (uint32_t i = start; i < cuda::std::min(start + 8U, count); ++i) {
                uint64_t forward = 0, reverse = 0;
                bool valid = true;
                for (uint32_t j = 0; valid && j < k; ++j) {
                    auto const at = tile + i + j;
                    // Codes as the tiled path assigns them: A 0, C 1, T 2, G 3; complement ^ 2.
                    uint32_t code = 0;
                    if (packed_input) {
                        code = (words[at / 16] >> (2 * (at % 16))) & 3U;
                    } else {
                        switch (sequence[at]) {
                            case 'A':
                            case 'a':
                                code = 0;
                                break;
                            case 'C':
                            case 'c':
                                code = 1;
                                break;
                            case 'T':
                            case 't':
                                code = 2;
                                break;
                            case 'G':
                            case 'g':
                                code = 3;
                                break;
                            default:
                                valid = false;
                        }
                    }
                    forward = (forward << 2) | code;
                    reverse |= uint64_t{code ^ 2U} << (2 * j);
                }
                if (!valid) continue;
                auto const word = forward > reverse ? forward : reverse;
                auto const hash = hash_kmer(word);
                if (static_cast<uint32_t>(hash >> 32) > hash_ceiling) continue;
                auto const incoming = score<Layout>(hash);
                auto const bucket = bucket_of<BucketCount>(hash);
                if constexpr (HasBlacklist) {
                    if (incoming > load_shared_relaxed(&local[bucket]) &&
                        !blacklist.contains(word, hash, bucket)) {
                        atomicMax(&local[bucket], incoming);
                    }
                } else {
                    atomicMax(&local[bucket], incoming);
                }
            }
        } else {
            // Four extra cells cover the k-1 halo, including the last partial cell.
            auto const padded = (count + 7) / 8 + 4;
            if (packed_input) {
                // Two-bit words hold sixteen bases, base j in bits 2j, and no ambiguous base. Tiles
                // start on a multiple of sixteen bases, so a tile's cells are whole halves of
                // words.
                auto const* const words = reinterpret_cast<uint32_t const*>(sequence);
                auto const total = (static_cast<size_t>(windows) + k - 1 + 7) / 8;
                for (uint32_t cell = threadIdx.x; cell < padded; cell += blockDim.x) {
                    auto const index = tile / 8 + cell;
                    if (index >= total) {
                        cells[cell] = missing_cell;
                        continue;
                    }
                    // A cell holds its first base in its highest bits: reverse the codes' order.
                    auto const reversed = __brev((words[index / 2] >> (16 * (index % 2))) << 16);
                    cells[cell] = ((reversed & 0x5555U) << 1) | ((reversed >> 1) & 0x5555U);
                }
            }
            for (uint32_t cell = threadIdx.x; !packed_input && cell < padded; cell += blockDim.x) {
                auto const pos = cell * 8;
                uint32_t ascii[2] = {};
                if ((reinterpret_cast<uintptr_t>(sequence + tile) & 7U) == 0 &&
                    pos + 7 < count + k - 1) {
                    auto const value = *reinterpret_cast<uint2 const*>(sequence + tile + pos);
                    ascii[0] = value.x;
                    ascii[1] = value.y;
                } else {
                    _Pragma("unroll")
                    for (uint32_t j = 0; j < 8; ++j) {
                        auto const byte = pos + j < count + k - 1
                                              ? static_cast<uint8_t>(sequence[tile + pos + j])
                                              : uint8_t{0xFF};
                        ascii[j / 4] |= static_cast<uint32_t>(byte) << (8 * (j % 4));
                    }
                }
                uint32_t packed = 0, bad = 0;
                _Pragma("unroll")
                for (uint32_t j = 0; j < 2; ++j) {
                    // Only the T code differs in ASCII bits 0 and 4.
                    auto const thymine = (ascii[j] & ~(ascii[j] << 1) & 0x04040404U) >> 2;
                    auto const normalised = (ascii[j] ^ (thymine * 0x11U)) & 0xD9D9D9D9U;
                    auto const bad_bytes = __vsetne4(normalised, 0x41414141U);
                    auto const codes = (ascii[j] >> 1) & 0x03030303U;
                    // Gather four codes in reverse byte order and four invalid flags in byte order.
                    packed = __dp4a(codes, 0x01041040U, packed << 8);
                    bad = __dp4a(bad_bytes, 0x08040201U << (4 * j), bad);
                }
                cells[cell] = packed | (bad << 16);
            }
            __syncthreads();
            auto const start = threadIdx.x * 8;
            if (start < count) {
                auto const cell = start / 8;
                auto const a = cells[cell], b = cells[cell + 1];
                auto const c = cells[cell + 2], d = cells[cell + 3];
                auto high =
                    (uint64_t{a} << 48) | (uint64_t{b & 0xFFFFU} << 32) | (c << 16) | (d & 0xFFFFU);
                uint64_t low = 0;
                auto bad = uint64_t{a >> 16} | (uint64_t{b >> 16} << 8) |
                           (uint64_t{c >> 16} << 16) | (uint64_t{d >> 16} << 24);
                if (k > 25) {
                    auto const e = cells[cell + 4];
                    low = uint64_t{e & 0xFFFFU} << 48;
                    bad |= uint64_t{e >> 16} << 32;
                }
                // Reverse the span once, then slide both strands across its eight windows.
                auto reverse_word = [](uint64_t word) {
                    auto const bits = cuda::std::bit_reverse(word);
                    return (((bits & 0xAAAAAAAAAAAAAAAAULL) >> 1) |
                            ((bits & 0x5555555555555555ULL) << 1)) ^
                           0xAAAAAAAAAAAAAAAAULL;
                };
                auto reverse_high = reverse_word(high);
                auto reverse_low = reverse_word(low);
                // An invalid base in the common overlap excludes all eight windows.
                auto const end = (bad & valid_mask & ~uint64_t{0x7F}) == 0
                                     ? cuda::std::min(8U, count - start)
                                     : 0U;
                // Specialising on k > 25 keeps the slide branch-free; the compiler does not
                // unswitch it once the blacklist lookup is present.
                auto slide = [&]<bool Long>() {
                    _Pragma("unroll")
                    for (uint32_t i = 0; i < end; ++i) {
                        if ((bad & valid_mask) == 0) {
                            auto const forward = high >> (64 - 2 * k);
                            auto const reverse = reverse_high & mask;
                            auto const word = forward > reverse ? forward : reverse;
                            auto const hash = hash_kmer(word);
                            if (static_cast<uint32_t>(hash >> 32) <= hash_ceiling) {
                                auto const incoming = score<Layout>(hash);
                                auto const bucket = bucket_of<BucketCount>(hash);
                                // Look up only scores that can still raise the CTA's register.
                                if constexpr (HasBlacklist) {
                                    if (incoming > load_shared_relaxed(&local[bucket]) &&
                                        !blacklist.contains(word, hash, bucket)) {
                                        atomicMax(&local[bucket], incoming);
                                    }
                                } else {
                                    atomicMax(&local[bucket], incoming);
                                }
                            }
                        }
                        // Eight overlapping windows fit in one 32-base word when k <= 25.
                        if constexpr (Long) {
                            high = (high << 2) | (low >> 62);
                            low <<= 2;
                            reverse_high = (reverse_high >> 2) | (reverse_low << 62);
                            reverse_low >>= 2;
                        } else {
                            high <<= 2;
                            reverse_high >>= 2;
                        }
                        bad >>= 1;
                    }
                };
                if (k > 25) {
                    slide.template operator()<true>();
                } else {
                    slide.template operator()<false>();
                }
            }
        }
        __syncthreads();
        // Share accepted winners across CTAs before processing the remaining tiles.
        if ((++tiles == 4 || tiles % 16 == 0) && tile + tile_stride < windows) {
            using reduce = cub::BlockReduce<uint32_t, 256>;
            __shared__ typename reduce::TempStorage exchange_scratch;
            __shared__ uint32_t exchange_ceiling;
            uint32_t minimum = 0xffffU;
            // Each thread owns its buckets here, so the merged value feeds the floor directly.
            for (uint32_t b = threadIdx.x; b < BucketCount; b += blockDim.x) {
                auto const value = local[b];
                auto global =
                    cuda::atomic_ref<uint32_t, cuda::thread_scope_device>{registers[b]}.load(
                        cuda::memory_order_relaxed
                    );
                if (value > global) global = cuda::std::max(value, atomicMax(&registers[b], value));
                local[b] = global;
                minimum = cuda::std::min(minimum, global);
            }
            auto const result = reduce(exchange_scratch).Reduce(minimum, cuda::minimum<>{});
            if (threadIdx.x == 0) {
                // An inclusive high-word cutoff admits ties for exact per-bucket pruning.
                exchange_ceiling =
                    result == 0 ? UINT32_MAX : uint32_t(restore<Layout>(result) >> 32);
            }
            __syncthreads();
            hash_ceiling = exchange_ceiling;
        }
    }
    __syncthreads();
    for (uint32_t i = threadIdx.x; i < BucketCount; i += blockDim.x) {
        if (local[i] != 0U) atomicMax(&registers[i], local[i]);
    }
}

agree()

Result< void > cuddl::detail::agree ( Result< void >  local, MPI_Comm  comm  )

inline

Returns local on the rank that failed, and on every other rank an error naming the lowest failing rank, so that all ranks leave a collective step together.

Definition at line 75 of file distributed.cuh.

                                                                           {
    int rank = 0, size = 1;
    CUDDL_TRY(mpi_try(MPI_Comm_rank(comm, &rank), "MPI_Comm_rank"));
    CUDDL_TRY(mpi_try(MPI_Comm_size(comm, &size), "MPI_Comm_size"));
    int const mine = local ? size : rank;
    int failed = size;
    CUDDL_TRY(mpi_try(MPI_Allreduce(&mine, &failed, 1, MPI_INT, MPI_MIN, comm), "MPI_Allreduce"));
    if (!local) return local;
    if (failed == size) return Ok();
    return Err(Error::resource("rank " + std::to_string(failed) + " failed"));
}

Here is the call graph for this function:

Here is the caller graph for this function:

align_up()

uintptr_t cuddl::detail::align_up ( uintptr_t  address, size_t  alignment  )

inlinenoexcept

Definition at line 944 of file reference_database.cuh.

                                                                                      {
    return (address + alignment - 1U) & ~(static_cast<uintptr_t>(alignment) - 1U);
}

Here is the caller graph for this function:

all_to_all_tile_queries()

constexpr uint32_t cuddl::detail::all_to_all_tile_queries ( uint32_t  reference_count, uint32_t  parts = 1U  )

constexprnoexcept

Query rows one all-to-all tile compares: about 2^25 pairs, and at least batch_query_tile_count.

Enough query groups to fill the GPU for each launch, while result storage holds two tiles so one is searched as the other is consumed.

Split into parts shards, tiles shrink until each part gets about four, so the alternating assignment evens out the triangle. They shrink only when the whole search already spans several tiles: its two-slot result storage then holds the smaller tiles too.

Definition at line 977 of file reference_database.cuh.

                                                                                {
    constexpr uint32_t pair_budget = 1U << 25U;
    constexpr uint64_t tiles_per_part = 4U;
    auto const limit = reference_count == 0U
                           ? batch_query_tile_count
                           : std::max(batch_query_tile_count, pair_budget / reference_count);
    auto const whole = batch_query_tile_size(reference_count, reference_count, limit);
    if (parts < 2U || whole == 0U || whole >= reference_count) return whole;
    auto const per_part = static_cast<uint32_t>(reference_count / (parts * tiles_per_part));
    return std::min(whole, std::max(batch_query_tile_count, per_part));
}

Here is the call graph for this function:

Here is the caller graph for this function:

available_device_bytes()

Result< size_t > cuddl::detail::available_device_bytes ( cuda::stream_ref  stream )

inline

Device bytes an allocation on stream can still get.

Storage the default pool caches from earlier allocations is reused by the next one, so it counts as available alongside what the device reports free.

Definition at line 127 of file database_staging.cuh.

                                                                                  {
    size_t free_bytes = 0, device_bytes = 0;
    CUDDL_CUDA_TRY(cudaMemGetInfo(&free_bytes, &device_bytes));
    auto const& device_pool = cuda::device_default_memory_pool(stream.device());
    auto const pool_reserved =
        device_pool.attribute(cuda::memory_pool_attributes::reserved_mem_current);
    auto const pool_used = device_pool.attribute(cuda::memory_pool_attributes::used_mem_current);
    return free_bytes + pool_reserved - std::min(pool_reserved, pool_used);
}

Here is the caller graph for this function:

batch_cardinality_kernel()

template<size_t BucketCount, typename Layout = default_register_layout>

__global__ void cuddl::detail::batch_cardinality_kernel	(	uint32_t const *const	registers,
		uint32_t const	row_count,
		uint64_t *const	empty_out,
		double *const	estimates_out
	)

Computes the cardinality of every stored sketch in one launch.

One block reduces one row of the row-major store of BucketCount registers per sketch.

Definition at line 2649 of file kernels.cuh.

  {
    auto const row = static_cast<size_t>(blockIdx.x);
    if (row >= row_count) {
        return;
    }
    using block_reduce = cub::BlockReduce<cardinality_payload, block_size>;
    __shared__ typename block_reduce::TempStorage storage;
    auto const* const mine = registers + row * BucketCount;
    cardinality_payload local{};
    for (auto bucket = static_cast<size_t>(threadIdx.x); bucket < BucketCount;
         bucket += blockDim.x) {
        auto const stored = static_cast<uint16_t>(__ldcs(&mine[bucket]));
        if (stored == 0U) {
            ++local.empty;
        } else {
            local.restored += static_cast<float>(restore_midpoint<Layout>(stored));
        }
    }
    auto const total = block_reduce(storage).Sum(local);
    if (threadIdx.x == 0) {
        empty_out[row] = total.empty;
        estimates_out[row] = static_cast<double>(cardinality_f32(
            static_cast<float>(BucketCount), static_cast<float>(total.empty), total.restored
        ));
    }
}

Here is the call graph for this function:

batch_query_tile_size()

constexpr uint32_t cuddl::detail::batch_query_tile_size ( uint32_t  reference_count, uint32_t  query_count, uint32_t  tile_limit = batch_query_tile_count  )

constexprnoexcept

Definition at line 957 of file reference_database.cuh.

           {
    if (reference_count == 0U || query_count == 0U) {
        return 0U;
    }
    auto const count_capacity = std::numeric_limits<uint32_t>::max() / reference_count;
    return std::min(query_count, std::min(tile_limit, std::max(1U, count_capacity)));
}

Here is the caller graph for this function:

batch_result_slot()

template<bool UpperTriangle>

__host__ __device__ uint64_t cuddl::detail::batch_result_slot ( uint32_t  query_index, uint32_t  reference_id, uint32_t  query_id_offset, uint32_t  reference_count  )

inlinenoexcept

Position of the pair (query_index of the tile, reference_id) in a tile's results: rows of reference_count references per query, or for all-to-all tiles the packed strict upper triangle of pairs with query_id < reference_id, where query IDs start at query_id_offset.

Definition at line 1751 of file kernels.cuh.

           {
    if constexpr (UpperTriangle) {
        auto const query = static_cast<uint64_t>(query_index);
        auto const preceding = query * (reference_count - query_id_offset - 1U) -
                               query * (query == 0U ? 0U : query - 1U) / 2U;
        return preceding + reference_id - (query_id_offset + query_index) - 1U;
    } else {
        return static_cast<uint64_t>(query_index) * reference_count + reference_id;
    }
}

batch_scores_kernel()

template<size_t BucketCount>

__global__ void cuddl::detail::batch_scores_kernel	(	uint32_t const *const	registers,
		uint32_t const	row_count,
		uint16_t *const	scores_out
	)

Copies the winning score of every register into compact row-major scores.

Definition at line 2683 of file kernels.cuh.

  {
    auto const row = static_cast<size_t>(blockIdx.x);
    if (row >= row_count) {
        return;
    }
    auto const* const mine = registers + row * BucketCount;
    auto* const scores = scores_out + row * BucketCount;
    for (auto bucket = static_cast<size_t>(threadIdx.x); bucket < BucketCount;
         bucket += blockDim.x) {
        scores[bucket] = static_cast<uint16_t>(__ldcs(&mine[bucket]));
    }
}

bbtools_mean_m()

__device__ double cuddl::detail::bbtools_mean_m ( double  bucket_count, double  filled, double  restored_sum  )

inlinenoexcept

BBTools occupancy-corrected MeanM estimate from restored magnitudes.

Definition at line 163 of file detail/hybrid_cardinality.cuh.

                                                                                 {
    if (filled <= 0.0 || restored_sum <= 0.0) {
        return 0.0;
    }
    auto const mean = restored_sum / filled;
    auto const occupancy_correction = (filled + bucket_count) / (2.0 * bucket_count);
    return hash_range / mean * filled * occupancy_correction;
}

Here is the caller graph for this function:

bbtools_mean_m_f32()

__device__ float cuddl::detail::bbtools_mean_m_f32 ( float  bucket_count, float  filled, float  restored_sum  )

inlinenoexcept

Single-precision bbtools_mean_m for the latency-bound hybrid scan.

Definition at line 305 of file detail/hybrid_cardinality.cuh.

                                                                                  {
    if (filled <= 0.0f || restored_sum <= 0.0f) {
        return 0.0f;
    }
    auto const mean = restored_sum / filled;
    auto const occupancy_correction = (filled + bucket_count) / (2.0f * bucket_count);
    return static_cast<float>(hash_range) / mean * filled * occupancy_correction;
}

Here is the caller graph for this function:

bgzf_block_size()

uint32_t cuddl::detail::bgzf_block_size ( unsigned char const *  header )

inlinenoexcept

A BGZF block's total size from its 18-byte header, or 0 if it is not one: a gzip header with only FEXTRA set, holding just the BC subfield.

Definition at line 76 of file device_gzip.cuh.

                                                                                    {
    unsigned char const expected[16] = {0x1f, 0x8b, 8, 4, 0, 0, 0, 0, 0, 0, 6, 0, 'B', 'C', 2, 0};
    for (size_t i = 0; i < sizeof(expected); ++i) {
        // Bytes 4-9 are MTIME, XFL and OS, which may hold anything.
        if ((i < 4 || i > 9) && header[i] != expected[i]) return 0;
    }
    return (uint32_t{header[16]} | uint32_t{header[17]} << 8U) + 1U;
}

Here is the caller graph for this function:

bucket_of()

template<size_t BucketCount>

__host__ __device__ constexpr size_t cuddl::detail::bucket_of ( uint64_t  hash )

constexprnoexcept

Selects the register bucket for hash given a power-of-two BucketCount.

Definition at line 27 of file hash.cuh.

                                                                       {
    static_assert(BucketCount != 0 && (BucketCount & (BucketCount - 1)) == 0);
    return hash & (BucketCount - 1);
}

cardinality()

__host__ __device__ double cuddl::detail::cardinality ( double  bucket_count, double  empty_count, double  sum_restored  )

inlinenoexcept

Public DDL cardinality estimate.

Minimum MLE is the current production estimator. Keeping this dispatch point makes changing the estimator explicit without exposing implementation-specific cardinality APIs.

Definition at line 59 of file cardinality.cuh.

                                                                                   {
    return minimum_mle(bucket_count, empty_count, sum_restored);
}

Here is the call graph for this function:

cardinality_f32()

__host__ __device__ float cuddl::detail::cardinality_f32 ( float  bucket_count, float  empty_count, float  sum_restored  )

inlinenoexcept

Single-precision cardinality for the throughput-bound cardinality reduction.

Definition at line 91 of file cardinality.cuh.

                                                                                    {
    return minimum_mle_f32(bucket_count, empty_count, sum_restored);
}

Here is the call graph for this function:

Here is the caller graph for this function:

cardinality_kernel()

template<size_t BucketCount, typename Layout = default_register_layout>

__global__ void cuddl::detail::cardinality_kernel	(	uint32_t const *	registers,
		uint64_t *	empty_out,
		double *	estimate_out
	)

Computes the cardinality of a single constructed sketch.

empty_out receives the empty-register count; estimate_out receives the estimate.

Definition at line 2551 of file kernels.cuh.

                                                                                         {
    using block_reduce = cub::BlockReduce<cardinality_payload, block_size>;
    __shared__ typename block_reduce::TempStorage storage;
    cardinality_payload local{};
    for (auto bucket = static_cast<size_t>(threadIdx.x); bucket < BucketCount;
         bucket += blockDim.x) {
        auto const stored = static_cast<uint16_t>(__ldcs(&registers[bucket]));
        if (stored == 0U) {
            ++local.empty;
        } else {
            local.restored += static_cast<float>(restore_midpoint<Layout>(stored));
        }
    }
    auto const total = block_reduce(storage).Sum(local);
    if (threadIdx.x == 0) {
        *empty_out = total.empty;
        *estimate_out = static_cast<double>(cardinality_f32(
            static_cast<float>(BucketCount), static_cast<float>(total.empty), total.restored
        ));
    }
}

Here is the call graph for this function:

classify()

__device__ void cuddl::detail::classify ( pairwise_counts &  target, uint32_t  this_score, uint32_t  other_score  )

inlinenoexcept

Classifies one score relative to this into target.

Takes 16-bit scores and 32-bit registers alike. Branchless: all four outcomes are computed as predicated arithmetic so a warp with mixed comparisons (random sketch data) never pays branch-reconvergence costs. Both-empty is the equality outcome restricted to zero scores.

Definition at line 16 of file comparison.cuh.

                                                                                      {
    auto const this_zero = static_cast<uint32_t>(this_score == 0U);
    auto const lt = static_cast<uint32_t>(this_score < other_score);
    auto const gt = static_cast<uint32_t>(this_score > other_score);
    auto const eq = 1U - lt - gt;
    target.both_empty += eq & this_zero;
    target.lower += lt;
    target.higher += gt;
    target.equal += eq & (1U - this_zero);
}

Here is the caller graph for this function:

classify_query_words()

template<typename ReferenceScore >

__device__ __forceinline__ void cuddl::detail::classify_query_words ( pairwise_counts &  target, uint32_t const (&)  q[8], ReferenceScore const *  reference  )

noexcept

Classifies 16 buckets of an already loaded 16-bit query chunk against one load_256_alignment aligned reference chunk.

Definition at line 1241 of file kernels.cuh.

           {
    auto const* qs = reinterpret_cast<uint16_t const*>(q);
    if constexpr (sizeof(ReferenceScore) == 2U) {
        uint32_t r[8];
        load_256_global_nc(reinterpret_cast<uint32_t const*>(reference), r);
        // Differences of zero-extended 16-bit scores carry the comparison in their sign bit,
        // and a sum of two scores minus one is negative only when both are empty. Equal is
        // whatever remains of the chunk's 16 buckets.
        uint32_t lower = 0U;
        uint32_t higher = 0U;
        uint32_t empty = 0U;
        _Pragma("unroll")
        for (uint32_t i = 0; i < 8U; ++i) {
            auto const q_low = q[i] & 0xffffU;
            auto const q_high = q[i] >> 16U;
            auto const r_low = r[i] & 0xffffU;
            auto const r_high = r[i] >> 16U;
            lower += ((q_low - r_low) >> 31U) + ((q_high - r_high) >> 31U);
            higher += ((r_low - q_low) >> 31U) + ((r_high - q_high) >> 31U);
            empty += ((q_low + r_low - 1U) >> 31U) + ((q_high + r_high - 1U) >> 31U);
        }
        target.lower += lower;
        target.higher += higher;
        target.both_empty += empty;
        target.equal += 16U - lower - higher - empty;
    } else {
        uint32_t r0[8];
        uint32_t r1[8];
        load_256_global_nc(reinterpret_cast<uint32_t const*>(reference), r0);
        load_256_global_nc(reinterpret_cast<uint32_t const*>(reference) + 8U, r1);
        _Pragma("unroll")
        for (uint32_t i = 0; i < 8U; ++i) {
            classify(target, qs[i], r0[i]);
        }
        _Pragma("unroll")
        for (uint32_t i = 0; i < 8U; ++i) {
            classify(target, qs[8U + i], r1[i]);
        }
    }
}

Here is the call graph for this function:

compact_fastx_sequence_extents()

void cuddl::detail::compact_fastx_sequence_extents ( fastx_sequence_file &  result )

inline

Compacts sequence whitespace in place, leaving every extent a run of bases.

Consumers stage extents verbatim and choose piece boundaries on any base, which needs contiguous bases. Compacting here moves the work onto the loader thread: each file is compacted while the other loaders are still decompressing, and a consumer that used to skip whitespace byte by byte now walks a dense span.

Definition at line 1413 of file fastx_sequence_file.hpp.

                                                                        {
    if (result.extents.empty()) return;
    // Extents point into whichever buffer holds the bytes: a page-locked target, the heap copy of
    // a compressed file, or the FASTQ sequence buffer. Those are writable; a memory-mapped plain
    // file is not, so its bases are compacted into `decompressed` instead.
    struct writable_buffer {
        char* data;
        size_t size;
    };
    auto const buffers = std::array{
        writable_buffer{result.decompressed_target, result.decompressed_size},
        writable_buffer{result.decompressed.data(), result.decompressed.size()},
        writable_buffer{result.sequence.data(), result.sequence.size()},
    };
    auto const writable = [&](char const* first, size_t size) {
        return std::any_of(buffers.begin(), buffers.end(), [&](writable_buffer const& buffer) {
            return buffer.data != nullptr && buffer.size >= size && first >= buffer.data &&
                   first + size <= buffer.data + buffer.size;
        });
    };
    size_t raw_total = 0;
    for (auto const& extent : result.extents)
        raw_total += static_cast<size_t>(extent.end - extent.begin);
    // Reserved up front so appending cannot move the buffer the extents are about to point at.
    std::string copied;
    copied.reserve(raw_total);
    std::vector<std::pair<size_t, std::pair<size_t, size_t>>> offsets;
    for (size_t index = 0; index < result.extents.size(); ++index) {
        auto const& extent = result.extents[index];
        auto const size = static_cast<size_t>(extent.end - extent.begin);
        if (writable(extent.begin, size)) {
            // Compaction only ever moves bytes towards the buffer's start.
            auto* const write = compact_sequence_whitespace(
                extent.begin, extent.end, const_cast<char*>(extent.begin)
            );
            result.extents[index] = {extent.begin, write};
        } else {
            auto const begin = copied.size();
            // Sized before the copy so the write needs no bounds check per byte.
            copied.resize(begin + size);
            auto* const write =
                compact_sequence_whitespace(extent.begin, extent.end, copied.data() + begin);
            copied.resize(static_cast<size_t>(write - copied.data()));
            offsets.emplace_back(index, std::pair{begin, copied.size()});
        }
    }
    if (!copied.empty()) {
        result.decompressed = std::move(copied);
        for (auto const& [index, span] : offsets) {
            result.extents[index] = {
                result.decompressed.data() + span.first, result.decompressed.data() + span.second
            };
        }
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

compact_sequence_whitespace()

char * cuddl::detail::compact_sequence_whitespace ( char const *  first, char const *  last, char *  out  )

inline

Copies bases from [first, last) to out, dropping sequence whitespace.

A sequence line is a long run of bases closed by a line ending, so a test per byte is the bulk of loading a corpus. Whole words are copied while none of their bytes is whitespace, and only a word that holds one falls back to bytes, which leaves the common case at one store per eight bases and keeps out no further along than the word being read.

Returns

The end of the written span.

Definition at line 1186 of file fastx_sequence_file.hpp.

                                                                                         {
    if (static_cast<size_t>(last - first) >= 64 && simd_compaction) {
#if defined(__ARM_NEON)
        return compact_sequence_whitespace_neon(first, last, out);
#elif defined(__x86_64__) || defined(__i386__)
        if (compact_avx512) return compact_sequence_whitespace_avx512(first, last, out);
        if (compact_avx2) {
            return compact_sequence_whitespace_avx2(first, last, out);
        }
#endif
    }
    constexpr uint64_t ones = 0x0101010101010101ULL;
    auto const has_zero_byte = [](uint64_t word) {
        return (word - ones) & ~word & 0x8080808080808080ULL;
    };
    auto const has_byte = [&](uint64_t word, uint64_t byte) {
        return has_zero_byte(word ^ (ones * byte));
    };
    auto const* word = first;
    while (static_cast<size_t>(last - word) >= sizeof(uint64_t)) {
        uint64_t value = 0;
        std::memcpy(&value, word, sizeof(value));
        auto const whitespace = has_byte(value, ' ') | has_byte(value, '\t') |
                                has_byte(value, '\n') | has_byte(value, '\r');
        if (whitespace == 0) {
            std::memcpy(out, word, sizeof(value));
            out += sizeof(value);
        } else {
            for (size_t index = 0; index < sizeof(value); ++index) {
                if (!fastx_is_sequence_whitespace(word[index])) *out++ = word[index];
            }
        }
        word += sizeof(value);
    }
    for (auto const* at = word; at != last; ++at) {
        if (!fastx_is_sequence_whitespace(*at)) *out++ = *at;
    }
    return out;
}

Here is the call graph for this function:

Here is the caller graph for this function:

compare_plane_group()

template<bool CountMatches, bool ReferenceFull = false>

__device__ __forceinline__ void cuddl::detail::compare_plane_group ( uint32_t const (&)  q[score_planes], uint32_t const (&)  r[score_planes], uint32_t  reference_any, uint32_t  reference_low_any, uint16_t  key_mask, uint32_t  match_lanes, plane_counts &  counts  )

noexcept

Adds one group's comparison of query planes q with reference planes r, whose nonzero buckets reference_any the caller hoists out of per-query loops.

From the least significant plane up, each plane's verdict overrides the lower planes' unless its bits are equal: one 3-input logic operation per plane for lt, one for ne. With count_matches, buckets whose masked keys agree and whose scores are both nonzero are counted too, as the index counts them.

With CountMatches, matches are counted in the buckets of match_lanes; the two stored key masks come from the plane chain: a full key matches exactly the equal nonempty buckets, and the folded 15-bit key those whose low planes agree and that are nonempty in both rows, which needs the reference's low-plane nonzero mask reference_low_any. Only those two masks pass score_compatibility validation.

ReferenceFull asserts every reference score is nonzero, so no bucket is both empty.

Definition at line 1457 of file kernels.cuh.

           {
    uint32_t lt = 0U;
    uint32_t ne = 0U;
    uint32_t low_ne = 0U;
    _Pragma("unroll")
    for (uint32_t p = 0; p < score_planes; ++p) {
        if (p == score_planes - 1U) {
            low_ne = ne;
        }
        lt = (~q[p] & r[p]) | (~(q[p] ^ r[p]) & lt);
        ne |= q[p] ^ r[p];
    }
    counts.lower += static_cast<uint32_t>(__popc(lt));
    counts.differ += static_cast<uint32_t>(__popc(ne));
    if constexpr (!ReferenceFull) {
        counts.occupied += static_cast<uint32_t>(__popc(ne | reference_any));
    }
    if constexpr (CountMatches) {
        // Validated key masks cover the low planes and, for a full key, the top one. Where the
        // compared planes agree, the low planes agree, so both rows are nonzero exactly when the
        // shared low planes are or both top bits are set.
        constexpr auto top = score_planes - 1U;
        auto const top_keyed = 0U - (static_cast<uint32_t>(key_mask) >> top);
        auto const key_differs = (ne & top_keyed) | (low_ne & ~top_keyed);
        counts.matches += static_cast<uint32_t>(
            __popc(~key_differs & (reference_low_any | (q[top] & r[top])) & match_lanes)
        );
    }
}

Here is the call graph for this function:

constexpr()

if cuddl::detail::constexpr ( CounterBits !  = 0U )

constexpr

Definition at line 968 of file kernels.cuh.

                                     {
        // Each lane sums one word's bitmaps; half warps then expand two words at a time, so the
        // counter stores stay free of bank conflicts.
        auto const tile_words = (width + warp_width - 1U) / warp_width;
        auto const count = bitmap_counts[query_index];
        auto const* starts =
            bitmap_starts + static_cast<size_t>(query_index) * indexed_bucket_count;
        for (auto base = warp * warp_width; base < tile_words; base += blockDim.x) {
            auto const word = low / warp_width + base + lane;
            uint32_t counters[CounterBits]{};
            uint32_t passed = ~0U;
            if (base + lane < tile_words) {
                // References on or below the diagonal count as passed so that summing can stop
                // early; the flush excludes them.
                uint32_t excluded = 0U;
                if (all_to_all && word * warp_width <= query_id_offset + query_index) {
                    auto const skipped = query_id_offset + query_index - word * warp_width + 1U;
                    excluded = skipped >= warp_width ? ~0U : (1U << skipped) - 1U;
                }
                passed = sum_posting_bitmaps(
                    bitmaps,
                    starts,
                    count,
                    word,
                    minimum_matches,
                    minimum_matches == 0U ? ~0U : excluded,
                    counters
                );
            }
            auto const shift = (lane % 16U) * 2U;
            for (uint32_t pair = 0U; pair < warp_width / 2U; ++pair) {
                auto const source = 2U * pair + lane / 16U;
                uint32_t packed = 0U;
                _Pragma("unroll")
                for (uint32_t plane = 0U; plane < CounterBits; ++plane) {
                    auto const bits =
                        (__shfl_sync(0xffffffffU, counters[plane], source) >> shift) & 3U;
                    packed |= ((bits & 1U) | ((bits & 2U) << 15U)) << plane;
                }
                auto const pass = __shfl_sync(0xffffffffU, passed, source) >> shift;
                if (pass & 1U) {
                    packed = (packed & 0xffff0000U) | minimum_matches;
                }
                if (pass & 2U) {
                    packed = (packed & 0xffffU) | (minimum_matches << 16U);
                }
                auto const slot = (base + source) * 16U + lane % 16U;
                if (slot < (width + 1U) / 2U) {
                    packed_counts[slot] = packed;
                }
            }
        }
    } else {

Here is the call graph for this function:

consume_byte()

void cuddl::detail::consume_byte ( char  ch, uint32_t  k, uint64_t  mask, span_result &  result, uint64_t &  window, uint32_t &  window_len  )

inline

Feeds one byte of sequence through the DDL rolling-window extraction.

Definition at line 236 of file fasta_parser.hpp.

  {
    if (fastx_is_sequence_whitespace(ch)) {
        return;
    }
    auto const encoded = encode_base(ch);
    if (encoded == 0xffu) {
        if (window_len < k && window_len > 0) {
            ++result.invalid_windows;
        }
        window_len = 0;
        window = 0;
        return;
    }
    ++result.bases;
    window = ((window << 2U) | encoded) & mask;
    if (window_len < k) {
        ++window_len;
    }
    if (window_len == k) {
        auto const rev = reverse_complement(window, k);
        result.kmers.push_back(window > rev ? window : rev);
        ++result.valid_kmers;
    }
}

Here is the call graph for this function:

crc32_combine()

uint32_t cuddl::detail::crc32_combine ( uint32_t  a, uint32_t  b, uint64_t  length  )

inlinenoexcept

CRC32 of A followed by B, from CRC32(A), CRC32(B) and B's length, as zlib's crc32_combine: CRC32(A) times x^(8 * length) modulo the reflected polynomial, plus CRC32(B).

Definition at line 87 of file device_gzip.cuh.

                                                                                              {
    constexpr uint32_t polynomial = 0xedb88320U;
    auto const multiply = [](uint32_t x, uint32_t y) {
        uint32_t product = 0;
        for (uint32_t bit = 1U << 31U; bit != 0; bit >>= 1U) {
            if ((x & bit) != 0) product ^= y;
            y = (y & 1U) != 0 ? (y >> 1U) ^ polynomial : y >> 1U;
        }
        return product;
    };
    // power = x^(2^k) for bit k of 8 * length; x^1 is 1 << 30 in reflected order.
    uint32_t shift = 1U << 31U, power = 1U << 30U;
    for (uint64_t bits = length * 8; bits != 0; bits >>= 1U) {
        if ((bits & 1U) != 0) shift = multiply(power, shift);
        power = multiply(power, power);
    }
    return multiply(shift, a) ^ b;
}

Here is the caller graph for this function:

database_file_little_endian()

template<typename T >

T cuddl::detail::database_file_little_endian ( T  value )

inline

Definition at line 40 of file reference_database_file.cuh.

                                              {
    if constexpr (std::endian::native == std::endian::little) {
        return value;
    } else {
        T result = 0;
        for (size_t i = 0; i < sizeof(T); ++i) {
            result = static_cast<T>((result << 8U) | (value & 0xffU));
            value >>= 8U;
        }
        return result;
    }
}

Here is the caller graph for this function:

database_file_metadata()

template<typename IO >

Result< void > cuddl::detail::database_file_metadata	(	IO &	io,
		reference_database_metadata &	metadata,
		uint32_t &	blacklist_count
	)

Definition at line 135 of file reference_database_file.cuh.

                                                                                                 {
    auto& c = metadata.compatibility;
    CUDDL_TRY(io.value(c.kmer_length));
    CUDDL_TRY(io.value(c.exponent_bits));
    CUDDL_TRY(io.value(c.bucket_count));
    CUDDL_TRY(io.value(c.key_mask));
    CUDDL_TRY(io.value(metadata.reference_count));
    CUDDL_TRY(io.value(blacklist_count));
    CUDDL_TRY(io.value(c.blacklist_identity));
    return Ok();
}

Here is the call graph for this function:

Here is the caller graph for this function:

decode_fastx_sequence_file()

Result< std::unique_ptr< fastx_sequence_file > > cuddl::detail::decode_fastx_sequence_file ( std::unique_ptr< fastx_mapped_file >  file, std::string const &  path, decompression_source  source, bool  defer_fasta, uint32_t  pack_k = 0  )

inline

Definition at line 1670 of file fastx_sequence_file.hpp.

  {
    auto result = std::make_unique<fastx_sequence_file>();
    result->file = std::move(file);
    auto data = result->file->data();
    auto& decompressed = result->decompressed;
    if (data.size() >= 2 && static_cast<unsigned char>(data[0]) == 0x1f &&
        static_cast<unsigned char>(data[1]) == 0x8b) {
        if (pack_k >= 16 && !defer_fasta && source.acquire != nullptr) {
            auto const fastq = CUDDL_TRY(inflate_and_pack(*result, data, path, source, pack_k));
            if (!fastq) return result;
            return finish_fastx_sequence_file(std::move(result), *fastq, path, false, pack_k);
        }
        // A single-member file's ISIZE is its inflated length. Asking for that rather than an
        // over-estimate keeps loader buffers small, which matters most when they are
        // page-locked, since the driver pins one allocation at a time. ISIZE sizes
        // nothing for a BGZF stream, which ends in an empty EOF block that reads back as zero, so
        // a hint below the compressed size falls back to the over-estimate the heap path assumes.
        // A concatenated stream advertises only its last member; a short target grows the
        // request, which keeps a multi-member file off the heap path.
        auto const hint = gzip_size_hint(data);
        auto request = hint >= data.size() ? hint : data.size() * 8;
        auto target = source.request(request);
        bool insufficient = false;
        bool inflated = false;
        size_t produced = 0;
        while (target.data != nullptr && target.capacity > 0) {
            auto const written = gunzip_members_direct(data, target, insufficient, path);
            if (!written) return Err(written.error());
            produced = *written;
            if (!insufficient) {
                inflated = true;
                break;
            }
            request = std::max(request * 2, data.size());
            target = source.request(request);
        }
        if (inflated) {
            result->decompressed_target = target.data;
            result->decompressed_size = produced;
            result->storage_owner = std::move(target.owner);
            data = std::string_view{target.data, produced};
        }
        if (result->decompressed_target == nullptr) {
            auto inflated = gunzip_members_into(data, decompressed, path);
            if (!inflated) return Err(inflated.error());
            data = decompressed;
        }
    } else if (source.acquire != nullptr && !defer_fasta) {
        // Plain bytes go to the caller's buffer, as inflated bytes do: the parser compacts them
        // in place and the stager reads them without another copy.
        auto target = source.request(data.size());
        if (target.data != nullptr) {
            std::memcpy(target.data, data.data(), data.size());
            result->decompressed_target = target.data;
            result->decompressed_size = data.size();
            result->storage_owner = std::move(target.owner);
            data = std::string_view{target.data, data.size()};
        }
    }
    return finish_fastx_sequence_file(std::move(result), data, path, defer_fasta, pack_k);
}

Here is the call graph for this function:

decompress_gzip_member()

libdeflate_result cuddl::detail::decompress_gzip_member ( libdeflate_decompressor *  decoder, std::string_view  input, char *  output, size_t  capacity, size_t &  consumed, size_t &  produced  )

inline

Decodes one gzip member, checking structure and ISIZE but not checksums.

Skipping the payload CRC avoids a second pass over every decompressed byte.

Definition at line 836 of file fastx_sequence_file.hpp.

  {
    auto const byte = [&](size_t i) {
        return static_cast<uint32_t>(static_cast<unsigned char>(input[i]));
    };
    if (input.size() < 18 || byte(0) != 0x1f || byte(1) != 0x8b || byte(2) != 8 ||
        (byte(3) & 0xe0) != 0) {
        return LIBDEFLATE_BAD_DATA;
    }
    auto const flags = byte(3);
    auto const end = input.size() - 8;
    size_t at = 10;
    if ((flags & 4) != 0) {
        if (end - at < 2) return LIBDEFLATE_BAD_DATA;
        auto const length = byte(at) | (byte(at + 1) << 8);
        at += 2;
        if (length > end - at) return LIBDEFLATE_BAD_DATA;
        at += length;
    }
    for (auto const flag : {8U, 16U}) {
        if ((flags & flag) != 0) {
            auto const terminator = input.find('\0', at);
            if (terminator == std::string_view::npos || terminator >= end) {
                return LIBDEFLATE_BAD_DATA;
            }
            at = terminator + 1;
        }
    }
    if ((flags & 2) != 0) {
        if (end - at < 2) return LIBDEFLATE_BAD_DATA;
        at += 2;
    }
    size_t deflate_bytes = 0;
    auto const status = libdeflate_deflate_decompress_ex(
        decoder, input.data() + at, end - at, output, capacity, &deflate_bytes, &produced
    );
    if (status != LIBDEFLATE_SUCCESS) return status;
    auto const trailer = at + deflate_bytes;
    auto const size = byte(trailer + 4) | (byte(trailer + 5) << 8) | (byte(trailer + 6) << 16) |
                      (byte(trailer + 7) << 24);
    if (static_cast<uint32_t>(produced) != size) return LIBDEFLATE_BAD_DATA;
    consumed = trailer + 8;
    return LIBDEFLATE_SUCCESS;
}

Here is the caller graph for this function:

device_pinned_pool()

cuda::pinned_memory_pool_ref cuddl::detail::device_pinned_pool ( cuda::device_ref  device, pinned_placement  placement  )

inline

A page-locked memory pool accessible from device only.

CCCL's default pinned pool is accessible from every visible device, so every allocation is mapped into all of them. With one MPI rank per GPU and all 8 GPUs of an A40 node visible to every rank, a 4-8 MB allocation took 27 ms instead of 1.3 ms, and sketch loaders spent most of their time waiting on the driver. Downloads into it with 8 ranks took 1.3 s, against 0.39 s from a device-node pool.

One pool per device and placement for the life of the process, like CUDA's default pools: never destroying it keeps it valid for buffers that live until process exit.

Definition at line 40 of file pinned_pool.cuh.

                                                                      {
    static std::mutex mutex;
    static std::vector<std::array<cudaMemPool_t, 2>> pools(cuda::devices.size());
    std::lock_guard lock(mutex);
    auto& pool = pools.at(static_cast<size_t>(device.get()))[static_cast<size_t>(placement)];
    if (pool == nullptr) {
        // Ablated, the pool behaves like CCCL's default pinned pool: mapped into every visible
        // device and returning freed pages to the OS at every synchronisation. numa-pool
        // separately puts device_node pages on any node.
        bool const ablate = ablated("pinned-pool");
        int node = -1;
        CUDDL_CUDA_CALL(cudaDeviceGetAttribute(&node, cudaDevAttrHostNumaId, device.get()));
        bool const numa =
            !ablated("numa-pool") && placement == pinned_placement::device_node && node >= 0 &&
            device.attribute(cuda::device_attributes::host_numa_memory_pools_supported) != 0;
        cudaMemPoolProps properties{};
        properties.allocType = cudaMemAllocationTypePinned;
        properties.location = numa ? cudaMemLocation{cudaMemLocationTypeHostNuma, node}
                                   : cudaMemLocation{cudaMemLocationTypeHost, 0};
        cudaMemPool_t created = nullptr;
        CUDDL_CUDA_CALL(cudaMemPoolCreate(&created, &properties));
        cudaMemAccessDesc const access{
            {cudaMemLocationTypeDevice, device.get()}, cudaMemAccessFlagsProtReadWrite
        };
        CUDDL_CUDA_CALL(cudaMemPoolSetAccess(created, &access, 1));
        if (ablate) {
            for (auto const& other : cuda::devices) {
                cudaMemAccessDesc const shared{
                    {cudaMemLocationTypeDevice, other.get()}, cudaMemAccessFlagsProtReadWrite
                };
                CUDDL_CUDA_CALL(cudaMemPoolSetAccess(created, &shared, 1));
            }
        }
        // Pinning is the expensive part of an allocation: with the default threshold of zero,
        // every synchronisation returns freed pages to the OS and the next allocation pins them
        // again, which cost sketch loaders more time than inflating their files.
        auto keep = ablate ? uint64_t{0} : std::numeric_limits<uint64_t>::max();
        CUDDL_CUDA_CALL(cudaMemPoolSetAttribute(created, cudaMemPoolAttrReleaseThreshold, &keep));
        pool = created;
    }
    return cuda::pinned_memory_pool_ref(pool);
}

Here is the call graph for this function:

Here is the caller graph for this function:

device_reads_pageable_memory()

bool cuddl::detail::device_reads_pageable_memory ( cuda::device_ref  device )

inlinenoexcept

Whether device reads pageable host memory itself.

This is what makes a CPU/GPU system coherent, Grace Hopper and Grace Blackwell among them, and what lets a kernel read the loader's buffer instead of a copy of it.

Definition at line 80 of file database_staging.cuh.

                                                                                       {
    return device.attribute(cuda::device_attributes::pageable_memory_access) != 0 &&
           device.attribute(
               cuda::device_attributes::pageable_memory_access_uses_host_page_tables
           ) != 0;
}

Here is the caller graph for this function:

directory_posting_range()

__device__ uint2 cuddl::detail::directory_posting_range ( uint16_t const *  sorted_keys, uint32_t const *  directory, uint32_t  reference_count, uint32_t  bucket, uint32_t  sparse_key  )

inline

One sparse bucket/key posting range, narrowed through the bucket's key directory.

Definition at line 505 of file kernels.cuh.

  {
    auto const* first = sorted_keys + static_cast<size_t>(bucket) * reference_count;
    auto const* cells =
        directory + static_cast<size_t>(bucket) * sparse_directory_entries(reference_count);
    auto const cell = sparse_key >> sparse_directory_shift(reference_count);
    auto const* low = first + cells[cell];
    auto const* high = first + cells[cell + 1U];
    auto const less = [](uint16_t key, uint32_t value) {
        return key < value;
    };
    auto const* begin = cuda::std::lower_bound(low, high, sparse_key, less);
    auto const* end = cuda::std::lower_bound(begin, high, sparse_key + 1U, less);
    return {static_cast<uint32_t>(begin - sorted_keys), static_cast<uint32_t>(end - sorted_keys)};
}

Here is the call graph for this function:

dlc()

__device__ double cuddl::detail::dlc ( uint32_t const *  bins, double  bucket_count, double  lc  )

inlinenoexcept

Distributed linear-counting estimate, blended from per-tier estimates by precision.

Definition at line 98 of file detail/hybrid_cardinality.cuh.

                                                                                            {
    auto const empty = bins[0];
    auto const minimum = max(1.0, bucket_count * dlc_min_fraction);
    auto const maximum = bucket_count - minimum;
    auto cumulative = empty;
    double weight_sum = 0.0;
    double weighted_log_sum = 0.0;
    for (int32_t tier = 0; tier + 1 < nlz_bins; ++tier) {
        if (cumulative >= minimum && cumulative <= maximum) {
            auto const occupied = bucket_count - cumulative;
            auto const log_ratio = cuda::std::log(bucket_count / cumulative);
            auto const tier_estimate = cuda::std::ldexp(bucket_count * log_ratio, tier);
 
            auto const error = cuda::std::sqrt(2.0 / 3.14159265358979323846) *
                               cuda::std::sqrt(occupied / (bucket_count * cumulative)) / log_ratio;
 
            // `1 / error` is positive here. `pow(x, 4.5)` is the same value as
            // `x^4 * sqrt(x)` and avoids the generic double-precision pow path.
            auto const inv_error = 1.0 / error;
            auto const inv_squared = inv_error * inv_error;
            auto const weight = inv_squared * inv_squared * cuda::std::sqrt(inv_error);
            weight_sum += weight;
            weighted_log_sum += weight * cuda::std::log(tier_estimate);
        }
        cumulative += bins[tier + 1];
        if (cumulative >= bucket_count) {
            break;
        }
    }
    if (weight_sum == 0.0) {
        return lc;
    }
    auto const pure = cuda::std::exp(weighted_log_sum / weight_sum);
    if (empty >= dlc_blend_high * bucket_count) {
        return lc;
    }
    if (empty > dlc_blend_low * bucket_count) {
        auto const t = (empty - dlc_blend_low * bucket_count) /
                       ((dlc_blend_high - dlc_blend_low) * bucket_count);
        return t * lc + (1.0 - t) * pure;
    }
    return pure;
}

Here is the caller graph for this function:

dlc_f32()

__device__ float cuddl::detail::dlc_f32 ( uint32_t const *  bins, float  bucket_count, float  lc  )

inlinenoexcept

Single-precision dlc for the latency-bound hybrid scan.

Definition at line 239 of file detail/hybrid_cardinality.cuh.

                                                                                             {
    auto const empty = static_cast<float>(bins[0]);
    auto const minimum = max(1.0f, bucket_count * static_cast<float>(dlc_min_fraction));
    auto const maximum = bucket_count - minimum;
    auto cumulative = empty;
    float weight_sum = 0.0f;
    float weighted_log_sum = 0.0f;
    for (int32_t tier = 0; tier + 1 < nlz_bins; ++tier) {
        if (cumulative >= minimum && cumulative <= maximum) {
            auto const occupied = bucket_count - cumulative;
            auto const log_ratio = cuda::std::log(bucket_count / cumulative);
            auto const tier_estimate = cuda::std::ldexp(bucket_count * log_ratio, tier);
 
            auto const error = cuda::std::sqrt(2.0f / 3.14159265358979323846f) *
                               cuda::std::sqrt(occupied / (bucket_count * cumulative)) / log_ratio;
 
            // `1 / error` is positive here. `pow(x, 4.5)` is the same value as
            // `x^4 * sqrt(x)` and avoids a generic pow path.
            auto const inv_error = 1.0f / error;
            auto const inv_squared = inv_error * inv_error;
            auto const weight = inv_squared * inv_squared * cuda::std::sqrt(inv_error);
            weight_sum += weight;
            weighted_log_sum += weight * cuda::std::log(tier_estimate);
        }
        cumulative += bins[tier + 1];
        if (cumulative >= bucket_count) {
            break;
        }
    }
    if (weight_sum == 0.0f) {
        return lc;
    }
    auto const pure = cuda::std::exp(weighted_log_sum / weight_sum);
    if (empty >= static_cast<float>(dlc_blend_high) * bucket_count) {
        return lc;
    }
    if (empty > static_cast<float>(dlc_blend_low) * bucket_count) {
        auto const t = (empty - static_cast<float>(dlc_blend_low) * bucket_count) /
                       ((static_cast<float>(dlc_blend_high) - static_cast<float>(dlc_blend_low)) *
                        bucket_count);
        return t * lc + (1.0f - t) * pure;
    }
    return pure;
}

Here is the caller graph for this function:

download_store()

template<size_t BucketCount>

Result< void > cuddl::detail::download_store	(	device_span< uint32_t const >	store,
		std::span< uint16_t >	scores,
		cuda::stream_ref	stream
	)

Copies the winner scores of device store rows into host score rows.

Scores are extracted on the device, so only two bytes per bucket cross the bus. scores must stay alive until stream reaches this point.

Definition at line 844 of file database_staging.cuh.

  {
    auto const genomes = scores.size() / BucketCount;
    if (genomes == 0) return Ok();
    auto device_scores =
        cuda::make_device_buffer<uint16_t>(stream, stream.device(), scores.size(), cuda::no_init);
    batch_scores_kernel<BucketCount>
        <<<static_cast<uint32_t>(genomes), block_size, 0, stream.get()>>>(
            store.data(), static_cast<uint32_t>(genomes), device_scores.data()
        );
    CUDDL_CUDA_TRY(cudaGetLastError());
    CUDDL_CUDA_TRY(cuda::copy_bytes(stream, device_scores, scores));
    return Ok();
}

Here is the call graph for this function:

encode_base()

constexpr uint8_t cuddl::detail::encode_base ( char  base )

inlineconstexprnoexcept

Maps one DNA byte to its 2-bit symbol (A=0, C=1, T=2, G=3) or 0xff when invalid.

Case-insensitive; any ambiguous byte (including 'N') is invalid. The __host__ __device__ annotation only exists under the CUDA compiler so plain host translation units (e.g. via cuddl/fastx.hpp) see a normal inline function.

Definition at line 16 of file dna.hpp.

                                                         {
    auto const byte = static_cast<uint8_t>(base);
    auto const upper = static_cast<uint8_t>(byte & 0xDFu);
    auto const x = (byte >> 1u) & 3u;
    auto const valid =
        static_cast<uint8_t>((upper == 'A') | (upper == 'C') | (upper == 'G') | (upper == 'T'));
    auto const mask = static_cast<uint8_t>(0u - valid);
    return static_cast<uint8_t>((x & mask) | (0xFFu & ~mask));
}

Here is the caller graph for this function:

external_batch_query_limit()

uint32_t cuddl::detail::external_batch_query_limit ( uint32_t  reference_count, cuda::device_ref  device  )

inline

Query rows one external batch tile may hold on device.

A tile's results, match counts and counters cost about 20 bytes per pair, and a quarter of the device's memory goes to them, so a 16 GB device holds about 200 million pairs per tile. Larger tiles give the counting kernels more queries to overlap. The limit depends only on the device, so storage sized from the requirements always matches the search.

Definition at line 996 of file reference_database.cuh.

                                                                            {
    constexpr size_t bytes_per_pair = sizeof(packed_pairwise_counts) + 3U * sizeof(uint32_t);
    auto const budget = device.attribute(cuda::device_attributes::total_global_memory) / 4U;
    auto const pairs = budget / bytes_per_pair;
    auto const queries = reference_count == 0U ? pairs : pairs / reference_count;
    return static_cast<uint32_t>(
        std::clamp<size_t>(queries, batch_query_tile_count, std::numeric_limits<uint32_t>::max())
    );
}

fasta_next_header()

char const * cuddl::detail::fasta_next_header ( char const *  first, char const *  last  )

inline

First '>' in [first, last) that starts a line, or last.

The byte before first must be a line ending.

Definition at line 1229 of file fastx_sequence_file.hpp.

                                                                                        {
    for (auto const* at = first; at != last; ++at) {
        at = static_cast<char const*>(std::memchr(at, '>', static_cast<size_t>(last - at)));
        if (at == nullptr) return last;
        if (at == first || at[-1] == '\n' || at[-1] == '\r') return at;
    }
    return last;
}

Here is the caller graph for this function:

fasta_sequence_compact()

char * cuddl::detail::fasta_sequence_compact ( char const *  first, char const *  last, char *  out, char const *&  stop  )

inline

Compacts one FASTA record's sequence from first to out, stopping at the next header.

The byte before first must be a line ending, and out may alias first. stop receives the next line-start '>' or last.

Returns

The end of the written span.

Definition at line 1366 of file fastx_sequence_file.hpp.

                                                                                          {
#if (defined(__x86_64__) || defined(__i386__)) && (defined(__GNUC__) || defined(__clang__))
    if (compact_avx512) return fasta_sequence_compact_avx512(first, last, out, stop);
    if (compact_avx2) return fasta_sequence_compact_avx2(first, last, out, stop);
#endif
#if defined(__ARM_NEON)
    if (simd_compaction) return fasta_sequence_compact_neon(first, last, out, stop);
#endif
    stop = fasta_next_header(first, last);
    return compact_sequence_whitespace(first, stop, out);
}

Here is the call graph for this function:

Here is the caller graph for this function:

fastx_fasta_compact_extents()

std::vector< fastx_sequence_extent > cuddl::detail::fastx_fasta_compact_extents ( std::string_view  data, char *  out  )

inline

Collects FASTA sequence extents with whitespace compacted out, in one pass over data.

An extent spans the bases between one line-start '>' header line and the next header (or the end of data); bytes before the first header belong to no extent. Bases of consecutive records are packed back to back from out, which may alias data.

Definition at line 1384 of file fastx_sequence_file.hpp.

                                                            {
    std::vector<fastx_sequence_extent> extents;
    size_t header = data.find('>');
    while (header != std::string_view::npos && header != 0 && data[header - 1] != '\n' &&
           data[header - 1] != '\r') {
        header = data.find('>', header + 1);
    }
    if (header == std::string_view::npos) return extents;
    auto const* const last = data.data() + data.size();
    while (true) {
        auto const end = fastx_line_end(data, header);
        if (end + 1 >= data.size()) break;
        auto const* const sequence = data.data() + end + 1;
        char const* stop = nullptr;
        auto* const begin = out;
        out = fasta_sequence_compact(sequence, last, out, stop);
        if (stop != sequence) extents.push_back({begin, out});
        if (stop == last) break;
        header = static_cast<size_t>(stop - data.data());
    }
    return extents;
}

Here is the call graph for this function:

Here is the caller graph for this function:

fastx_is_sequence_whitespace()

constexpr bool cuddl::detail::fastx_is_sequence_whitespace ( char  ch )

inlineconstexprnoexcept

True for bytes that FASTA sequence consumers conventionally skip between bases.

Definition at line 61 of file fastx_sequence_file.hpp.

                                                                                   {
    return ch == '\n' || ch == '\r' || ch == ' ' || ch == '\t';
}

Here is the caller graph for this function:

fastx_line_end()

size_t cuddl::detail::fastx_line_end ( std::string_view  data, size_t  position  )

inline

First line-end position at or after position using the resolved scanner.

Definition at line 195 of file fastx_sequence_file.hpp.

                                                                                 {
    return line_end(data, position);
}

Here is the caller graph for this function:

fastx_line_end_scalar()

size_t cuddl::detail::fastx_line_end_scalar ( std::string_view  data, size_t  position  )

inline

Definition at line 92 of file fastx_sequence_file.hpp.

                                                                                        {
    while (position < data.size() && data[position] != '\n' && data[position] != '\r') {
        ++position;
    }
    return position;
}

Here is the caller graph for this function:

fastx_split_sequence_spans()

std::vector< fastx_sequence_span > cuddl::detail::fastx_split_sequence_spans ( std::vector< fastx_sequence_extent > const &  extents, uint32_t  seed_bases, uint32_t  count  )

inline

Splits the combined FASTA sequence stream into count contiguous spans.

Each span carries the raw byte ranges of its slice of the sequence stream plus a materialised prefix of the preceding seed_bases valid bases (whitespace included, headers excluded, cut short at an invalid base or the buffer start). A rolling-window consumer that replays the prefix, then consumes the segments, reproduces the single-threaded byte stream exactly, so the per-span results can be concatenated in order. Whitespace between bases is preserved in both the prefix and the segments; consumers skip it with fastx_is_sequence_whitespace.

Definition at line 122 of file fasta_parser.hpp.

  {
    std::vector<fastx_sequence_span> spans;
    if (extents.empty() || count == 0U) {
        return spans;
    }
    size_t total = 0;
    for (auto const& extent : extents) {
        total += static_cast<size_t>(extent.end - extent.begin);
    }
    if (total == 0) {
        return spans;
    }
    if (static_cast<size_t>(count) > total) {
        count = static_cast<uint32_t>(total);
    }
    auto const per = total / count + (total % count != 0U);
    spans.reserve(count);
 
    for (uint32_t t = 0; t < count; ++t) {
        auto const span_begin = static_cast<size_t>(t) * per;
        if (span_begin >= total) {
            break;
        }
        auto const span_end = span_begin + per < total ? span_begin + per : total;
 
        fastx_sequence_span span;
        span.total_bytes = span_end - span_begin;
 
        // Locate the extent and offset of the span start.
        size_t extent = 0;
        size_t offset = span_begin;
        while (extent < extents.size() &&
               offset >= static_cast<size_t>(extents[extent].end - extents[extent].begin)) {
            offset -= static_cast<size_t>(extents[extent].end - extents[extent].begin);
            ++extent;
        }
 
        // Seed: walk back up to seed_bases valid bases (or until an invalid base / EOF).
        size_t seed_extent = extent;
        char const* seed_start = extents[extent].begin + offset;
        {
            size_t e = extent;
            char const* q = seed_start;
            uint32_t collected = 0;
            while (collected < seed_bases && span_step_back(extents, e, q)) {
                auto const ch = *q;
                if (fastx_is_sequence_whitespace(ch)) {
                    continue;
                }
                if (encode_base(ch) == 0xffu) {
                    seed_extent = e;
                    seed_start = q;
                    span_step_forward(extents, seed_extent, seed_start);
                    break;
                }
                ++collected;
                seed_extent = e;
                seed_start = q;
            }
        }
 
        // Materialise the prefix from the seed start to the span start.
        {
            size_t e = seed_extent;
            char const* q = seed_start;
            char const* target = extents[extent].begin + offset;
            while (e < extent || q < target) {
                if (q >= extents[e].end) {
                    ++e;
                    q = extents[e].begin;
                    continue;
                }
                span.prefix.push_back(*q);
                ++q;
            }
        }
 
        // Collect the span's segments.
        {
            size_t remaining = span_end - span_begin;
            char const* q = extents[extent].begin + offset;
            size_t e = extent;
            while (remaining > 0 && e < extents.size()) {
                auto const limit = static_cast<size_t>(extents[e].end - q);
                auto const step = limit < remaining ? limit : remaining;
                span.segments.emplace_back(q, step);
                remaining -= step;
                if (remaining > 0) {
                    ++e;
                    if (e < extents.size()) {
                        q = extents[e].begin;
                    }
                }
            }
        }
 
        spans.push_back(std::move(span));
    }
    return spans;
}

Here is the call graph for this function:

finalize()

constexpr __host__ __device__ __forceinline__ uint64_t cuddl::detail::finalize ( uint64_t  h )

constexpr

Applies the xxHash-64 final mixing (avalanche) step.

Definition at line 53 of file hash.cuh.

                                                                            {
    h ^= h >> 33;
    h *= PRIME64_2;
    h ^= h >> 29;
    h *= PRIME64_3;
    h ^= h >> 32;
    return h;
}

Here is the caller graph for this function:

finish_fastx_sequence_file()

Result< std::unique_ptr< fastx_sequence_file > > cuddl::detail::finish_fastx_sequence_file ( std::unique_ptr< fastx_sequence_file >  result, std::string_view  data, std::string const &  path, bool  defer_fasta, uint32_t  pack_k  )

inline

Parses, packs or defers the bytes data of a loaded file, which result holds.

Definition at line 1575 of file fastx_sequence_file.hpp.

  {
    result->input = data;
    if (defer_fasta) {
        auto const first = data.find_first_not_of("\r\n");
        if (first != std::string_view::npos && data[first] == '>') {
            auto const end = fastx_line_end(data, first);
            // Empty first records and malformed inputs stay with the CPU parser.
            if (end + 1 < data.size() && data[end + 1] != '>') return result;
        }
    }
    if (pack_k >= 16) {
        auto const first = data.find_first_not_of("\r\n");
        if (first == std::string_view::npos || data[first] != '@') {
            char* out = result->decompressed_target;
            if (out == nullptr) {
                if (data.data() != result->decompressed.data())
                    result->decompressed.resize(data.size());
                out = result->decompressed.data();
            }
            CUDDL_TRY(pack_fasta_chunks(*result, data, out, pack_k, path));
            return result;
        }
    }
    CUDDL_TRY(parse_fastx_sequence_file(*result, path));
    return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

for()

cuddl::detail::for

(

)

Definition at line 444 of file kernels.cuh.

                                                           {
        auto const bucket = i / entries;
        auto const bound = static_cast<uint32_t>(i % entries) << shift;
        auto const* first = sorted_keys + bucket * reference_count;
        directory[i] = static_cast<uint32_t>(
            cuda::std::lower_bound(
                first,
                first + reference_count,
                bound,
                [](uint16_t key, uint32_t value) { return key < value; }
            ) -
            first
        );
    }

gunzip_members_direct()

Result< size_t > cuddl::detail::gunzip_members_direct ( std::string_view  input, decompression_target  target, bool &  insufficient, std::string const &  path  )

inline

Decompresses gzip members straight into a caller's buffer.

Returns Error::resource when target is too small, which callers treat as a request to fall back to the growing-string path rather than a failure.

Definition at line 931 of file fastx_sequence_file.hpp.

  {
    insufficient = false;
    static thread_local shared_gzip_decompressor shared;
    if (shared.handle == nullptr) {
        return Err(Error::resource("cannot allocate gzip decompressor"));
    }
    size_t consumed = 0;
    size_t produced = 0;
    while (consumed < input.size()) {
        size_t member_in = 0;
        size_t member_out = 0;
        auto const status = decompress_gzip_member(
            shared.handle,
            input.substr(consumed),
            target.data + produced,
            target.capacity - produced,
            member_in,
            member_out
        );
        if (status == LIBDEFLATE_INSUFFICIENT_SPACE) {
            // A caller-supplied buffer cannot grow, so the loader falls back instead.
            insufficient = true;
            return Result<size_t>::ok(0);
        }
        if (status != LIBDEFLATE_SUCCESS) {
            return Err(Error::invalid_argument("gzip FASTX error: invalid stream: " + path));
        }
        consumed += member_in;
        produced += member_out;
    }
    return produced;
}

Here is the call graph for this function:

Here is the caller graph for this function:

gunzip_members_into()

Result< void > cuddl::detail::gunzip_members_into ( std::string_view  input, std::string &  output, std::string const &  path  )

inline

Decompresses gzip members from a mapped buffer into output, appending.

Members are decompressed one at a time so concatenated streams (BGZF among them) work. Each member's structure, DEFLATE data and length trailer are checked, but CRC32 is not. output grows as needed, so pre-sizing storage saves reallocations without truncating data.

Definition at line 893 of file fastx_sequence_file.hpp.

                                                                                    {
    static thread_local shared_gzip_decompressor shared;
    if (shared.handle == nullptr) {
        return Err(Error::resource("cannot allocate gzip decompressor"));
    }
    // Genomes compress around 3x; guessing low costs a realloc per member, not correctness.
    if (output.empty()) output.resize(std::max<size_t>(input.size() * 4, 1U << 16));
    size_t consumed = 0;
    size_t produced = 0;
    while (consumed < input.size()) {
        size_t member_in = 0;
        size_t member_out = 0;
        auto const status = decompress_gzip_member(
            shared.handle,
            input.substr(consumed),
            output.data() + produced,
            output.size() - produced,
            member_in,
            member_out
        );
        if (status == LIBDEFLATE_INSUFFICIENT_SPACE) {
            output.resize(output.size() * 2 + 1);
            continue;
        }
        if (status != LIBDEFLATE_SUCCESS) {
            return Err(Error::invalid_argument("gzip FASTX error: invalid stream: " + path));
        }
        consumed += member_in;
        produced += member_out;
    }
    output.resize(produced);
    return Ok();
}

Here is the call graph for this function:

Here is the caller graph for this function:

gzip_decompressed_size()

uint64_t cuddl::detail::gzip_decompressed_size ( std::string const &  path )

inline

Uncompressed size the gzip trailer advertises, or 0 when it cannot be read.

ISIZE holds one member's length modulo 2^32, so a concatenated stream is only sized correctly by its last member. A short buffer is refused later, never silently truncated.

Exact decompressed size of path, or 0 when it is not a readable gzip file.

A gzip member ends with a little-endian ISIZE, so this needs a 4-byte read instead of a full decompression pass. A corpus of these bounds how much a staged arena can ever hold.

Definition at line 977 of file fastx_sequence_file.hpp.

                                                                            {
    auto file = std::fopen(path.c_str(), "rb");
    if (file == nullptr) return 0;
    uint64_t size = 0;
    if (std::fseek(file, -4, SEEK_END) == 0) {
        unsigned char trailer[4] = {};
        if (std::fread(trailer, 1, sizeof(trailer), file) == sizeof(trailer)) {
            size = static_cast<uint64_t>(trailer[0]) | (static_cast<uint64_t>(trailer[1]) << 8) |
                   (static_cast<uint64_t>(trailer[2]) << 16) |
                   (static_cast<uint64_t>(trailer[3]) << 24);
        }
    }
    std::fclose(file);
    return size;
}

Here is the caller graph for this function:

gzip_size_hint()

size_t cuddl::detail::gzip_size_hint ( std::string_view  input )

inline

Definition at line 993 of file fastx_sequence_file.hpp.

                                                                 {
    if (input.size() < 8) return 0;
    auto const byte = [&](size_t index) {
        return static_cast<uint32_t>(static_cast<unsigned char>(input[index]));
    };
    auto const tail = input.size() - 4;
    return byte(tail) | (byte(tail + 1) << 8) | (byte(tail + 2) << 16) | (byte(tail + 3) << 24);
}

Here is the caller graph for this function:

has_gzip_signature_portable()

bool cuddl::detail::has_gzip_signature_portable ( char const *  data, size_t  size  )

inline

True when data holds what could start another gzip member: 1f 8b 08, then a flag byte with its reserved bits clear.

A member with reserved flag bits set fails to decode, so it cannot be one the host path would inflate. Compressed data holds the three-byte signature alone in about one file in eleven, which would send those files back to the host.

Definition at line 221 of file device_gzip.cuh.

                                                                                     {
    for (size_t at = 0; size - at >= 4;) {
        auto const* const found =
            static_cast<char const*>(::memmem(data + at, size - at - 1, "\x1f\x8b\x08", 3));
        if (found == nullptr) return false;
        if ((static_cast<uint8_t>(found[3]) & 0xe0U) == 0) return true;
        at = static_cast<size_t>(found - data) + 1;
    }
    return false;
}

Here is the caller graph for this function:

hash_kmer()

__host__ __device__ constexpr uint64_t cuddl::detail::hash_kmer ( uint64_t  packed_kmer )

constexprnoexcept

Hashes a canonical packed k-mer into the value space used by DDL registers.

Definition at line 21 of file hash.cuh.

                                                                                {
    return splitmix64(packed_kmer ^ seed);
}

Here is the call graph for this function:

hybrid()

__device__ double cuddl::detail::hybrid ( double  low_estimate, double  zone_estimate, double  corrected_mean, double  bucket_count  )

inlinenoexcept

Log-blended low and corrected estimates across the hybrid transition zone.

Definition at line 143 of file detail/hybrid_cardinality.cuh.

           {
    auto const low = hybrid_low * bucket_count;
    auto const high = hybrid_high * bucket_count;
    if (zone_estimate <= low) {
        return low_estimate;
    }
    if (zone_estimate >= high) {
        return corrected_mean;
    }
    auto const t = cuda::std::log(zone_estimate / low) / cuda::std::log(high / low);
    return (1.0 - t) * low_estimate + t * corrected_mean;
}

hybrid_cardinality_kernel()

template<size_t BucketCount, typename Layout = default_register_layout>

__global__ void cuddl::detail::hybrid_cardinality_kernel	(	uint32_t const *const	registers,
		hybrid_cardinality_estimates *const	estimates
	)

Computes BBTools and paper-style HybridDDL estimates in one register scan.

Definition at line 2575 of file kernels.cuh.

  {
    __shared__ uint32_t bins[nlz_bins];
    using block_histogram =
        cub::BlockHistogram<uint32_t, block_size, 1, nlz_bins, cub::BLOCK_HISTO_ATOMIC>;
    using block_reduce = cub::BlockReduce<float, block_size>;
    __shared__ typename block_reduce::TempStorage storage;
    block_histogram histogram;
    histogram.InitHistogram(bins);
    float local_restored = 0.0f;
    __syncthreads();
 
    for (auto bucket = static_cast<size_t>(threadIdx.x); bucket < BucketCount;
         bucket += blockDim.x) {
        auto const stored = static_cast<uint16_t>(__ldcs(&registers[bucket]));
        uint32_t bin[1]{
            stored == 0U ? 0U : static_cast<uint32_t>(stored >> Layout::mantissa_bits) + 1U
        };
        histogram.Composite(bin, bins);
        if (stored != 0U) {
            local_restored += static_cast<float>(restore<Layout>(stored));
        }
    }
    __syncthreads();
    auto const restored = block_reduce(storage).Sum(local_restored);
    if (threadIdx.x == 0) {
        *estimates = hybrid_estimates_f32(bins, static_cast<float>(BucketCount), restored);
    }
}

Here is the call graph for this function:

hybrid_cardinality_variant_kernel()

template<size_t BucketCount, hybrid_variant Variant, typename Layout = default_register_layout>

__global__ void cuddl::detail::hybrid_cardinality_variant_kernel	(	uint32_t const *const	registers,
		double *const	estimate
	)

Computes one HybridDDL variant estimate for estimator comparisons.

Definition at line 2610 of file kernels.cuh.

                                                                                           {
    __shared__ uint32_t bins[nlz_bins];
    using block_histogram =
        cub::BlockHistogram<uint32_t, block_size, 1, nlz_bins, cub::BLOCK_HISTO_ATOMIC>;
    using block_reduce = cub::BlockReduce<float, block_size>;
    __shared__ typename block_reduce::TempStorage storage;
    block_histogram histogram;
    histogram.InitHistogram(bins);
    float local_restored = 0.0f;
    __syncthreads();
 
    for (auto bucket = static_cast<size_t>(threadIdx.x); bucket < BucketCount;
         bucket += blockDim.x) {
        auto const stored = static_cast<uint16_t>(__ldcs(&registers[bucket]));
        uint32_t bin[1]{
            stored == 0U ? 0U : static_cast<uint32_t>(stored >> Layout::mantissa_bits) + 1U
        };
        histogram.Composite(bin, bins);
        if (stored != 0U) {
            local_restored += static_cast<float>(restore<Layout>(stored));
        }
    }
    __syncthreads();
    auto const restored = block_reduce(storage).Sum(local_restored);
    if (threadIdx.x == 0) {
        auto const estimates =
            hybrid_estimates_f32(bins, static_cast<float>(BucketCount), restored);
        if constexpr (Variant == hybrid_variant::bbtools) {
            *estimate = estimates.bbtools;
        } else {
            *estimate = estimates.paper;
        }
    }
}

Here is the call graph for this function:

hybrid_estimates()

__device__ hybrid_cardinality_estimates cuddl::detail::hybrid_estimates ( uint32_t const *  bins, double  bucket_count, double  restored_sum  )

inlinenoexcept

Fused BBTools, paper, linear-counting, DLC, and raw MeanM estimates.

Definition at line 174 of file detail/hybrid_cardinality.cuh.

                                                                                          {
    auto const empty = static_cast<double>(bins[0]);
    auto const filled = bucket_count - empty;
    auto const raw_mean = bbtools_mean_m(bucket_count, filled, restored_sum);
    auto const lc = lc_min(bins, bucket_count);
    auto const dlc_estimate = dlc(bins, bucket_count, lc);
    auto const corrected_mean = raw_mean * interpolate_mean_m_cf(dlc_estimate);
    return {
        hybrid(lc, lc, corrected_mean, bucket_count),
        hybrid(lc, dlc_estimate, corrected_mean, bucket_count),
        lc,
        dlc_estimate,
        raw_mean,
    };
}

Here is the call graph for this function:

hybrid_estimates_f32()

__device__ hybrid_cardinality_estimates cuddl::detail::hybrid_estimates_f32 ( uint32_t const *  bins, float  bucket_count, float  restored_sum  )

inlinenoexcept

Single-precision hybrid_estimates for the latency-bound hybrid scan.

Definition at line 316 of file detail/hybrid_cardinality.cuh.

                                                                                            {
    auto const empty = static_cast<float>(bins[0]);
    auto const filled = bucket_count - empty;
    auto const raw_mean = bbtools_mean_m_f32(bucket_count, filled, restored_sum);
    auto const lc = lc_min_f32(bins, bucket_count);
    auto const dlc_estimate = dlc_f32(bins, bucket_count, lc);
    auto const corrected_mean = raw_mean * interpolate_mean_m_cf_f32(dlc_estimate);
    return {
        static_cast<double>(hybrid_f32(lc, lc, corrected_mean, bucket_count)),
        static_cast<double>(hybrid_f32(lc, dlc_estimate, corrected_mean, bucket_count)),
        static_cast<double>(lc),
        static_cast<double>(dlc_estimate),
        static_cast<double>(raw_mean),
    };
}

Here is the call graph for this function:

Here is the caller graph for this function:

hybrid_f32()

__device__ float cuddl::detail::hybrid_f32 ( float  low_estimate, float  zone_estimate, float  corrected_mean, float  bucket_count  )

inlinenoexcept

Single-precision hybrid for the latency-bound hybrid scan.

Definition at line 285 of file detail/hybrid_cardinality.cuh.

           {
    auto const low = static_cast<float>(hybrid_low) * bucket_count;
    auto const high = static_cast<float>(hybrid_high) * bucket_count;
    if (zone_estimate <= low) {
        return low_estimate;
    }
    if (zone_estimate >= high) {
        return corrected_mean;
    }
    auto const t = cuda::std::log(zone_estimate / low) / cuda::std::log(high / low);
    return (1.0f - t) * low_estimate + t * corrected_mean;
}

Here is the caller graph for this function:

hybrid_posting_ranges_kernel()

template<size_t BucketCount>

__global__ void cuddl::detail::hybrid_posting_ranges_kernel	(	uint16_t const *	queries,
		uint32_t const *	offsets,
		uint32_t const *	bitmap_cells,
		uint32_t	bitmap_count,
		uint32_t	words,
		uint32_t	indexed_buckets,
		uint16_t	key_mask,
		uint2 *	ranges,
		uint32_t *	bitmap_starts,
		uint32_t *	bitmap_counts
	)

Splits each query's posting ranges into short lists and bitmap offsets.

With no dense offsets, consumes sparse ranges already staged in ranges. Packs bitmap offsets in bucket order so nearby queries walk their shared bitmaps together.

Definition at line 775 of file kernels.cuh.

  {
    using block_scan = cub::BlockScan<uint32_t, block_size>;
    __shared__ typename block_scan::TempStorage storage;
    uint32_t count = 0U;
    auto const query = static_cast<uint32_t>(blockIdx.x);
    auto const base = static_cast<size_t>(query) * indexed_buckets;
    for (uint32_t first = 0U; first < indexed_buckets; first += block_size) {
        auto const bucket = first + threadIdx.x;
        uint2 range{0U, 0U};
        uint32_t bitmap = 0U;
        bool selected = false;
        auto const score = bucket < indexed_buckets
                               ? queries[static_cast<size_t>(query) * BucketCount + bucket]
                               : uint16_t{0};
        if (score != 0U) {
            auto const cell = bucket * (static_cast<uint32_t>(key_mask) + 1U) + (score & key_mask);
            if (offsets != nullptr) {
                range = {offsets[cell], offsets[cell + 1U]};
            } else {
                range = ranges[base + bucket];
            }
            if (range.y - range.x >= words) {
                uint32_t low = 0U, high = bitmap_count;
                while (low < high) {
                    auto const middle = low + (high - low) / 2U;
                    if (bitmap_cells[middle] < cell) {
                        low = middle + 1U;
                    } else {
                        high = middle;
                    }
                }
                bitmap = low * words;
                selected = true;
                range = {0U, 0U};
            }
        }
        uint32_t position = 0U, aggregate = 0U;
        block_scan(storage).ExclusiveSum(static_cast<uint32_t>(selected), position, aggregate);
        if (selected) {
            bitmap_starts[base + count + position] = bitmap;
        }
        if (bucket < indexed_buckets) {
            ranges[base + bucket] = range;
        }
        count += aggregate;
        __syncthreads();
    }
    if (threadIdx.x == 0U) {
        bitmap_counts[query] = count;
    }
}

Here is the call graph for this function:

if() [1/5]

cuddl::detail::if

(

block >=static_cast< uint64_t >(row_words) *

word_count

)

if() [2/5]

cuddl::detail::if

(

candidate_bits !

= nullptr && all_to_all && high <= query_id_offset + query_index + 1U

)

Definition at line 942 of file kernels.cuh.

                                                                                               {
        auto* bits = candidate_bits +
                     static_cast<size_t>(query_index) * candidate_bit_words(reference_count) +
                     low / warp_width;
        for (auto i = threadIdx.x; i < (high - low + warp_width - 1U) / warp_width;
             i += blockDim.x) {
            bits[i] = 0U;
        }
        return;
    }

if() [3/5]

cuddl::detail::if

(

candidate_bits

= = nullptr

)

Definition at line 1174 of file kernels.cuh.

                                   {
        auto* const counts =
            match_counts + static_cast<size_t>(query_index) * reference_count + low;
        for (auto i = threadIdx.x; i < width; i += blockDim.x) {
            counts[i] = (packed_counts[i / 2U] >> ((i % 2U) * 16U)) & 0xffffU;
        }
        return;
    }

if() [4/5]

cuddl::detail::if

(

local !

= 0U

)

Definition at line 499 of file kernels.cuh.

                     {
        atomicAdd(work, local);
    }

if() [5/5]

cuddl::detail::if

(

query >=tile.

query_count

)

index_tile_references()

uint32_t cuddl::detail::index_tile_references ( uint32_t  reference_count, uint32_t  query_count, uint32_t  multiprocessors, uint32_t  max_shared_bytes  )

inlinenoexcept

References per tile, bounded by the device's shared memory and small enough that the batch launches about four blocks per SM.

Tiles align to 64 references.

Definition at line 702 of file kernels.cuh.

           {
    constexpr uint32_t static_bytes = index_tile_long_capacity * sizeof(uint2) + sizeof(uint32_t);
    auto const max_references = (max_shared_bytes - static_bytes) / (64U * sizeof(uint16_t)) * 64U;
    auto const needed = (reference_count + max_references - 1U) / max_references;
    auto const filling = (4U * multiprocessors + query_count - 1U) / query_count;
    auto const tiles = needed > filling ? needed : filling;
    return ((reference_count + tiles - 1U) / tiles + 63U) / 64U * 64U;
}

indexed_cell_count()

constexpr uint64_t cuddl::detail::indexed_cell_count ( score_compatibility const &  compatibility )

inlineconstexprnoexcept

Definition at line 921 of file reference_database.cuh.

           {
    return static_cast<uint64_t>(compatibility.indexed_bucket_count) *
           (static_cast<uint64_t>(compatibility.key_mask) + 1U);
}

Here is the caller graph for this function:

indexed_posting_count()

constexpr uint64_t cuddl::detail::indexed_posting_count ( uint32_t  reference_count, score_compatibility const &  compatibility  )

inlineconstexprnoexcept

Definition at line 928 of file reference_database.cuh.

                                                                                                   {
    return static_cast<uint64_t>(reference_count) * compatibility.indexed_bucket_count;
}

Here is the caller graph for this function:

inflate_and_pack()

Result< std::optional< std::string_view > > cuddl::detail::inflate_and_pack ( fastx_sequence_file &  result, std::string_view  input, std::string const &  path, decompression_source  source, uint32_t  k  )

inline

Inflates gzip input in this thread's reusable heap buffer and, for FASTA, packs it there and moves only the packed words into a buffer from source.

The words are what crosses the bus, a quarter of the bases, so only they need page-locked memory. Pinning the whole inflated genome made pinning, which the driver runs one call at a time, what a fresh process with many loaders waited on. FASTQ is copied whole instead, and its bytes are returned for the parser; packed FASTA returns nothing.

Definition at line 1615 of file fastx_sequence_file.hpp.

  {
    thread_local std::unique_ptr<char[]> scratch;
    thread_local size_t capacity = 0;
    // A single-member file's ISIZE is its inflated length. ISIZE sizes nothing for a BGZF stream,
    // which ends in an empty EOF block that reads back as zero, so a hint below the compressed
    // size falls back to an over-estimate; a short buffer doubles and inflates again.
    auto const hint = gzip_size_hint(input);
    size_t wanted = hint >= input.size() ? hint : input.size() * 8;
    size_t produced = 0;
    while (true) {
        if (capacity < wanted) {
            capacity = std::bit_ceil(wanted);
            scratch = std::make_unique_for_overwrite<char[]>(capacity);
        }
        bool insufficient = false;
        produced = CUDDL_TRY(
            gunzip_members_direct(input, {scratch.get(), capacity, {}}, insufficient, path)
        );
        if (!insufficient) break;
        wanted = capacity * 2;
    }
    std::string_view const data{scratch.get(), produced};
    auto const first = data.find_first_not_of("\r\n");
    bool const fastq = first != std::string_view::npos && data[first] == '@';
    if (!fastq) CUDDL_TRY(pack_fasta_chunks(result, data, scratch.get(), k, path));
    uint64_t bytes = produced;
    if (!fastq) {
        uint64_t words = 0;
        for (auto const& record : result.packed) {
            words = std::max(words, record.word + (record.bases + 15) / 16);
        }
        bytes = words * sizeof(uint32_t);
    }
    auto target = source.request(bytes);
    char* out = target.data;
    if (out == nullptr) {
        result.decompressed.resize(bytes);
        out = result.decompressed.data();
    } else {
        result.decompressed_target = out;
        result.decompressed_size = bytes;
        result.storage_owner = std::move(target.owner);
    }
    std::memcpy(out, scratch.get(), bytes);
    if (fastq) return std::optional{std::string_view{out, bytes}};
    result.words = reinterpret_cast<uint32_t const*>(out);
    return std::optional<std::string_view>{};
}

Here is the call graph for this function:

Here is the caller graph for this function:

interpolate_mean_m_cf()

__device__ double cuddl::detail::interpolate_mean_m_cf ( double  estimate )

inlinenoexcept

Interpolates the MeanM correction factor for estimate from the calibration table.

Definition at line 59 of file detail/hybrid_cardinality.cuh.

                                                                         {
    if (estimate <= ddl_cf_keys[0]) {
        return ddl_mean_m_cf[0];
    }
    if (estimate >= ddl_cf_keys[ddl_cf_size - 1]) {
        return ddl_mean_m_cf[ddl_cf_size - 1];
    }
    uint32_t low = 0;
    uint32_t high = ddl_cf_size - 1;
    while (low + 1 < high) {
        auto const middle = (low + high) / 2;
        if (ddl_cf_keys[middle] <= estimate) {
            low = middle;
        } else {
            high = middle;
        }
    }
    auto const fraction = (estimate - ddl_cf_keys[low]) / (ddl_cf_keys[high] - ddl_cf_keys[low]);
    return ddl_mean_m_cf[low] + fraction * (ddl_mean_m_cf[high] - ddl_mean_m_cf[low]);
}

Here is the caller graph for this function:

interpolate_mean_m_cf_f32()

__device__ float cuddl::detail::interpolate_mean_m_cf_f32 ( float  estimate )

inlinenoexcept

Single-precision variant of the hybrid estimator tail.

The 2,048-register hybrid scan is memory/latency bound on every GPU, but the serial estimator tail in hybrid_estimates is FP64-heavy. Datacenter GPUs (H100/B200) execute those instructions at half FP32 rate, while consumer cards rate-limit FP64 to 1/64 of FP32. Keeping the scan and tail in FP32 avoids that consumer cliff; the ~1e-7 relative error is far below the ~2% standard error of a 2,048-register sketch.

Definition at line 200 of file detail/hybrid_cardinality.cuh.

                                                                           {
    if (estimate <= ddl_cf_keys[0]) {
        return ddl_mean_m_cf[0];
    }
    if (estimate >= ddl_cf_keys[ddl_cf_size - 1]) {
        return ddl_mean_m_cf[ddl_cf_size - 1];
    }
    uint32_t low = 0;
    uint32_t high = ddl_cf_size - 1;
    while (low + 1 < high) {
        auto const middle = (low + high) / 2;
        if (ddl_cf_keys[middle] <= estimate) {
            low = middle;
        } else {
            high = middle;
        }
    }
    auto const fraction = (estimate - ddl_cf_keys[low]) / (ddl_cf_keys[high] - ddl_cf_keys[low]);
    return ddl_mean_m_cf[low] + fraction * (ddl_mean_m_cf[high] - ddl_mean_m_cf[low]);
}

Here is the caller graph for this function:

lane_posting_range()

__device__ uint2 cuddl::detail::lane_posting_range ( uint32_t const *  offsets, uint16_t const *  sorted_keys, uint32_t  reference_count, uint32_t  bucket, uint32_t  key, uint32_t  key_count  )

inline

One thread's posting range for a bucket/key cell of a dense or sparse index.

Definition at line 395 of file kernels.cuh.

  {
    if (sorted_keys == nullptr) {
        auto const cell = static_cast<size_t>(bucket) * key_count + key;
        return {offsets[cell], offsets[cell + 1U]};
    }
    auto const* first = sorted_keys + static_cast<size_t>(bucket) * reference_count;
    auto const sparse_key = static_cast<uint16_t>(key + (key_count == 32768U));
    auto const range = cuda::std::equal_range(first, first + reference_count, sparse_key);
    return {
        static_cast<uint32_t>(range.first - sorted_keys),
        static_cast<uint32_t>(range.second - sorted_keys)
    };
}

launch_construction()

template<size_t BucketCount, typename Layout = default_register_layout>

__host__ Result< void > cuddl::detail::launch_construction ( device_span< uint64_t const >  input, device_span< uint32_t >  registers, cuda::stream_ref  stream, cuda::std::optional< blacklist_view >  blacklist = cuda::std::nullopt  )

inline

Launches construction of a sketch through a CTA-local shared register array merged into registers (add_shared_kernel).

Definition at line 17 of file construction.cuh.

  {
    if (input.empty()) {
        return {};
    }
    return cuda_try([&] {
        auto const multiprocessors =
            stream.device().attribute(cuda::device_attributes::multiprocessor_count);
        auto const vector_input = (reinterpret_cast<uintptr_t>(input.data()) & 31U) == 0U;
        // Two CTAs per SM; the grid-stride loop covers the remaining inputs.
        auto const capacity = static_cast<size_t>(shared_construction_block_size) * 4U;
        auto const needed = input.size() / capacity + (input.size() % capacity != 0U);
        auto const blocks =
            static_cast<uint32_t>(cuda::std::min<size_t>(multiprocessors * 2U, needed));
        auto launch = [&]<bool Filter>() {
            add_shared_kernel<BucketCount, Layout, Filter>
                <<<blocks, shared_construction_block_size, 0, stream.get()>>>(
                    input.data(),
                    input.size(),
                    registers.data(),
                    vector_input,
                    blacklist.value_or(blacklist_view{})
                );
        };
        if (blacklist) {
            launch.template operator()<true>();
        } else {
            launch.template operator()<false>();
        }
        return cudaGetLastError();
    });
}

Here is the call graph for this function:

launch_sequence_add()

template<size_t BucketCount, typename Layout = default_register_layout>

__host__ Result< void > cuddl::detail::launch_sequence_add ( device_span< char const >  sequence, uint32_t  k, device_span< uint32_t >  registers, cuda::stream_ref  stream, cuda::std::optional< blacklist_view >  blacklist = cuda::std::nullopt  )

inline

Shared launch for one device-resident ASCII chunk.

Windows derive as size >= k ? size - k + 1 : 0; short input is a no-op. Window counts above UINT32_MAX are rejected instead of narrowing, so size must not exceed UINT32_MAX + k - 1. No hidden carry: the caller supplies any K-1 overlap explicitly.

Definition at line 375 of file sequence_encode.cuh.

  {
    if (k < 1 || k > 31) {
        return Err(Error::invalid_argument("invalid k-mer length for sequence add"));
    }
    size_t const size = sequence.size();
    if (size < k) {
        return Ok();
    }
    size_t const windows_size = size - k + 1;
    if (windows_size > std::numeric_limits<uint32_t>::max()) {
        return Err(Error::invalid_argument("sequence chunk exceeds 32-bit window capacity"));
    }
    auto const windows = static_cast<uint32_t>(windows_size);
    auto launch = [&]<bool Filter>() -> Result<void> {
        return cuda_try([&] {
            auto const kernel = add_sequence_single_kernel<BucketCount, Layout, Filter>;
            auto const multiprocessors =
                stream.device().attribute(cuda::device_attributes::multiprocessor_count);
            // Fill one resident wave, accounting for the GPU and this sketch's shared memory.
            if constexpr (Filter) {
                // Leave L1 room for the blacklist presence map; the default favors shared memory.
                auto const carveout = cudaFuncSetAttribute(
                    kernel, cudaFuncAttributePreferredSharedMemoryCarveout, 50
                );
                if (carveout != cudaSuccess) return carveout;
            }
            int blocks_per_sm = 0;
            auto const occupancy =
                cudaOccupancyMaxActiveBlocksPerMultiprocessor(&blocks_per_sm, kernel, 256, 0);
            if (occupancy != cudaSuccess) return occupancy;
            size_t const need = (static_cast<size_t>(windows) + 2047) / 2048;
            size_t const capacity = static_cast<size_t>(multiprocessors) * blocks_per_sm;
            size_t const blocks_size =
                capacity == 0 ? size_t{1} : (capacity < need ? capacity : need);
            auto const blocks = static_cast<uint32_t>(blocks_size);
            kernel<<<blocks, 256, 0, stream.get()>>>(
                sequence.data(), windows, k, registers.data(), blacklist.value_or(blacklist_view{})
            );
            return cudaGetLastError();
        });
    };
    return blacklist ? launch.template operator()<true>() : launch.template operator()<false>();
}

Here is the call graph for this function:

lc_min()

__device__ double cuddl::detail::lc_min ( uint32_t const *  bins, double  bucket_count  )

inlinenoexcept

Linear-counting estimate from the first occupied NLZ tier.

Definition at line 81 of file detail/hybrid_cardinality.cuh.

                                                                                    {
    auto cumulative = bins[0];
    if (cumulative > 0U) {
        return bucket_count * cuda::std::log(bucket_count / cumulative);
    }
    for (int32_t tier = 0; tier + 1 < nlz_bins; ++tier) {
        cumulative += bins[tier + 1];
        if (cumulative > 0U) {
            return cuda::std::ldexp(
                bucket_count * cuda::std::log(bucket_count / cumulative), tier + 1
            );
        }
    }
    return 0.0;
}

Here is the caller graph for this function:

lc_min_f32()

__device__ float cuddl::detail::lc_min_f32 ( uint32_t const *  bins, float  bucket_count  )

inlinenoexcept

Single-precision lc_min for the latency-bound hybrid scan.

Definition at line 222 of file detail/hybrid_cardinality.cuh.

                                                                                      {
    auto cumulative = bins[0];
    if (cumulative > 0U) {
        return bucket_count * cuda::std::log(bucket_count / cumulative);
    }
    for (int32_t tier = 0; tier + 1 < nlz_bins; ++tier) {
        cumulative += bins[tier + 1];
        if (cumulative > 0U) {
            return cuda::std::ldexp(
                bucket_count * cuda::std::log(bucket_count / cumulative), tier + 1
            );
        }
    }
    return 0.0f;
}

Here is the caller graph for this function:

load_256_alignment()

__host__ __device__ constexpr uint32_t cuddl::detail::load_256_alignment ( )

constexprnoexcept

Byte alignment load_256_global_nc requires on the compiled architecture.

sm_100 and later issue one 256-bit load per call, which needs 32-byte alignment. Earlier architectures issue two 128-bit loads, which need only 16 bytes, so a 16-byte-aligned row still takes the wide path there.

Definition at line 22 of file helpers.cuh.

                                                                     {
#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000
    return 32U;
#else
    return 16U;
#endif
}

Here is the caller graph for this function:

load_256_global_nc()

template<typename T >

__device__ __forceinline__ void cuddl::detail::load_256_global_nc	(	const T *	ptr,
		T *	out
	)

Loads 256 bits from global memory using the non-coherent cache path.

Template Parameters

T	Element type (uint32_t or uint64_t)

Parameters

ptr	Source pointer (must be aligned to load_256_alignment)
out	Output array (4 elements for uint64_t, 8 for uint32_t)

Definition at line 47 of file helpers.cuh.

                                                                         {
    static_assert(sizeof(T) == 4 || sizeof(T) == 8, "T must be uint32_t or uint64_t");
 
#if __CUDA_ARCH__ >= 1000
    if constexpr (sizeof(T) == 8) {
        asm volatile("ld.global.nc.v4.u64 {%0, %1, %2, %3}, [%4];"
                     : "=l"(out[0]), "=l"(out[1]), "=l"(out[2]), "=l"(out[3])
                     : "l"(ptr));
    } else {
        asm volatile("ld.global.nc.v8.u32 {%0, %1, %2, %3, %4, %5, %6, %7}, [%8];"
                     : "=r"(out[0]),
                       "=r"(out[1]),
                       "=r"(out[2]),
                       "=r"(out[3]),
                       "=r"(out[4]),
                       "=r"(out[5]),
                       "=r"(out[6]),
                       "=r"(out[7])
                     : "l"(ptr));
    }
#else
    if constexpr (sizeof(T) == 8) {
        asm volatile("ld.global.nc.v2.u64 {%0, %1}, [%2];" : "=l"(out[0]), "=l"(out[1]) : "l"(ptr));
        asm volatile("ld.global.nc.v2.u64 {%0, %1}, [%2];"
                     : "=l"(out[2]), "=l"(out[3])
                     : "l"(ptr + 2));
    } else {
        asm volatile("ld.global.nc.v4.u32 {%0, %1, %2, %3}, [%4];"
                     : "=r"(out[0]), "=r"(out[1]), "=r"(out[2]), "=r"(out[3])
                     : "l"(ptr));
        asm volatile("ld.global.nc.v4.u32 {%0, %1, %2, %3}, [%4];"
                     : "=r"(out[4]), "=r"(out[5]), "=r"(out[6]), "=r"(out[7])
                     : "l"(ptr + 4));
    }
#endif
}

Here is the caller graph for this function:

load_chunk()

template<typename T >

__host__ __device__ __forceinline__ T cuddl::detail::load_chunk	(	const uint8_t *	data,
		uint64_t	index
	)

Loads a chunk of type T from data at byte offset index*sizeof(T).

Definition at line 46 of file hash.cuh.

                                                                                      {
    T chunk;
    memcpy(&chunk, data + index * sizeof(T), sizeof(T));
    return chunk;
}

load_fastx_sequence_file()

Result< std::unique_ptr< fastx_sequence_file > > cuddl::detail::load_fastx_sequence_file ( std::string const &  path, decompression_source  source = {}, bool  defer_fasta = false, bool  read = false, uint32_t  pack_k = 0  )

inline

Definition at line 1738 of file fastx_sequence_file.hpp.

                                  {},
    bool defer_fasta = false,
    bool read = false,
    uint32_t pack_k = 0
) {
#if defined(__linux__)
    if (source.acquire != nullptr) {
        int const fd = ::open(path.c_str(), O_RDONLY);
        struct stat status{};
        if (fd == -1 || ::fstat(fd, &status) != 0 || status.st_size < 0) {
            if (fd != -1) ::close(fd);
            return Err(Error::invalid_argument("cannot open FASTX file: " + path));
        }
        auto const size = static_cast<size_t>(status.st_size);
        unsigned char magic[2] = {};
        bool const gzip = ::pread(fd, magic, sizeof(magic), 0) == sizeof(magic) &&
                          magic[0] == 0x1f && magic[1] == 0x8b;
        // Plain bytes go straight into the caller's buffer, as inflated bytes do: the parser
        // compacts them in place and the stager reads them without another copy.
        auto target = gzip ? decompression_target{} : source.request(size);
        if (target.data == nullptr) {
            auto file = fastx_mapped_file::adopt(fd, size, path, read, source.direct_reads);
            if (!file) return Err(Error::invalid_argument("cannot open FASTX file: " + path));
            return decode_fastx_sequence_file(std::move(*file), path, source, defer_fasta, pack_k);
        }
        if (!read_whole_file(fd, target.data, size, source.direct_reads, target.capacity)) {
            return Err(Error::invalid_argument("cannot open FASTX file: " + path));
        }
        auto result = std::make_unique<fastx_sequence_file>();
        result->decompressed_target = target.data;
        result->decompressed_size = size;
        result->storage_owner = std::move(target.owner);
        return finish_fastx_sequence_file(
            std::move(result), {target.data, size}, path, defer_fasta, pack_k
        );
    }
#endif
    auto file = fastx_mapped_file::load(path, read, source.direct_reads);
    if (!file) return Err(Error::invalid_argument("cannot open FASTX file: " + path));
    return decode_fastx_sequence_file(std::move(*file), path, source, defer_fasta, pack_k);
}

Here is the caller graph for this function:

load_plane_chunk()

__device__ __forceinline__ void cuddl::detail::load_plane_chunk	(	uint4 const *	row,
		uint32_t	group,
		uint32_t	chunk,
		uint32_t(&)	words[4]
	)

Loads plane chunk chunk (planes 4 chunk ... 4 chunk + 3) of group group of a bit-plane row (score_plane_index layout).

Definition at line 2101 of file kernels.cuh.

                                                                                         {
    auto const value = row[((group / 32U) * 4U + chunk) * 32U + group % 32U];
    words[0] = value.x;
    words[1] = value.y;
    words[2] = value.z;
    words[3] = value.w;
}

Here is the caller graph for this function:

load_plane_group()

__device__ __forceinline__ void cuddl::detail::load_plane_group	(	uint4 const *	row,
		uint32_t	group,
		uint32_t(&)	words[score_planes]
	)

Loads group group of one bit-plane row (score_plane_index layout).

Definition at line 1380 of file kernels.cuh.

                                                                                    {
    _Pragma("unroll")
    for (uint32_t c = 0; c < score_planes / 4U; ++c) {
        auto const v = row[((group / 32U) * 4U + c) * 32U + group % 32U];
        words[4U * c] = v.x;
        words[4U * c + 1U] = v.y;
        words[4U * c + 2U] = v.z;
        words[4U * c + 3U] = v.w;
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

load_shared_relaxed()

__device__ __forceinline__ uint32_t cuddl::detail::load_shared_relaxed

(

uint32_t const *

address

)

Relaxed CTA-scope load of a shared word that other threads update atomically.

cuda::atomic_ref emits a generic-address load here; the shared state space avoids the address conversion and the 64-bit address registers.

Definition at line 29 of file sequence_encode.cuh.

                                                                                 {
    uint32_t value;
    asm volatile("ld.relaxed.cta.shared.u32 %0, [%1];"
                 : "=r"(value)
                 : "r"(static_cast<uint32_t>(__cvta_generic_to_shared(address)))
                 : "memory");
    return value;
}

make_refine_cells()

template<bool UpperTriangle>

__device__ __forceinline__ refine_cells cuddl::detail::make_refine_cells	(	uint32_t	query_id_offset,
		uint32_t	query_count,
		uint32_t	group,
		uint32_t	words_per_query,
		uint32_t	max_block_words,
		uint32_t	waves
	)

Splits a tile into refine_cells, about waves cells per block when the tile allows, so that blocks claiming cells dynamically even out cells of uneven work.

Every query of an upper-triangle tile follows query_id_offset, so words wholly before it hold no pairs and get no cells; otherwise those empty cells would take most of the split for tiles far down the triangle.

Definition at line 1717 of file kernels.cuh.

  {
    auto const groups = (query_count + group - 1U) / group;
    auto const first_word =
        UpperTriangle ? cuda::std::min(query_id_offset / 32U, words_per_query) : 0U;
    auto const words = words_per_query - first_word;
    auto const blocks_per_group = cuda::std::max(1U, waves * gridDim.x / groups);
    auto const block_words = cuda::std::max(
        1U, cuda::std::min(max_block_words, (words + blocks_per_group - 1U) / blocks_per_group)
    );
    return {groups, first_word, block_words, groups * ((words + block_words - 1U) / block_words)};
}

match_filter_kernel()

template<size_t BucketCount, bool UpperTriangle, uint32_t QueryGroup>

__global__ void cuddl::detail::match_filter_kernel	(	uint32_t const *	query_planes,
		uint32_t	query_id_offset,
		uint32_t	query_count,
		uint32_t const *	reference_planes,
		uint32_t	reference_count,
		uint32_t	indexed_bucket_count,
		uint32_t	minimum_matches,
		uint32_t *	candidate_bits,
		uint32_t *	cell_counter
	)

Records in candidate_bits (candidate-bitmap layout) the pairs of the layout whose scores agree in their low match_filter_planes planes in at least minimum_matches buckets.

Every validated key mask covers those planes, so the count bounds a pair's index matches from above and the candidates include every pair meeting the threshold. On RefSeq sketches the bound passes barely more pairs than the exact count; a thresholded bitmap refinement then counts matches exactly.

Cells, query staging, and word claiming follow refine_batch_bitmap_kernel. A warp takes a claimed word's references four at a time and keeps their low planes in registers, so each query plane chunk read from shared memory serves four pairs.

Definition at line 2125 of file kernels.cuh.

  {
    static_assert(match_filter_supported<BucketCount>);
    constexpr uint32_t warp_width = 32;
    constexpr uint32_t group = QueryGroup;
    static_assert(group >= 1U && group <= warp_width);
    constexpr uint32_t row_words = static_cast<uint32_t>(BucketCount / 2U);
    constexpr uint32_t lane_groups = static_cast<uint32_t>(BucketCount / warp_width / warp_width);
    constexpr uint32_t max_block_words = static_cast<uint32_t>(cuda::std::max<size_t>(
        1U, refine_block_row_bytes / (BucketCount * sizeof(uint16_t)) / warp_width
    ));
    extern __shared__ uint4 filter_query_planes[];
    __shared__ uint32_t next_word;
    __shared__ uint32_t claimed_cell;
 
    auto const lane = static_cast<uint32_t>(threadIdx.x) % warp_width;
    auto const words_per_query = candidate_bit_words(reference_count);
    auto const cells = make_refine_cells<UpperTriangle>(
        query_id_offset, query_count, group, words_per_query, max_block_words, 1U
    );
    auto const groups = cells.groups;
    auto const indexed_groups = indexed_bucket_count / warp_width;
    // Unindexed buckets are seeded as differing, so they never count toward the bound.
    uint32_t excluded_lanes[lane_groups];
    _Pragma("unroll")
    for (uint32_t j = 0; j < lane_groups; ++j) {
        excluded_lanes[j] = j * warp_width + lane < indexed_groups ? 0U : ~0U;
    }
 
    for (auto cell = next_cell(cell_counter, blockIdx.x - gridDim.x, claimed_cell);
         cell < cells.cell_count;
         cell = next_cell(cell_counter, cell, claimed_cell)) {
        auto const first_query = (cell % groups) * group;
        auto const group_size = cuda::std::min(group, query_count - first_query);
        auto const first_word = cells.first_word + (cell / groups) * cells.block_words;
        auto const last_word = cuda::std::min(first_word + cells.block_words, words_per_query);
        auto const* group_planes = reinterpret_cast<uint4 const*>(
            query_planes + static_cast<size_t>(first_query) * row_words
        );
 
        __syncthreads();
        if (threadIdx.x == 0U) {
            next_word = first_word;
        }
        for (auto i = static_cast<uint32_t>(threadIdx.x); i < group_size * (row_words / 4U);
             i += match_filter_block_size) {
            filter_query_planes[i] = group_planes[i];
        }
        __syncthreads();
 
        for (;;) {
            uint32_t word = 0U;
            if (lane == 0U) {
                word = atomicAdd(&next_word, 1U);
            }
            word = __shfl_sync(0xffffffffU, word, 0);
            if (word >= last_word) {
                break;
            }
            // Lane m holds query m's selected references in this word.
            uint32_t bits = 0U;
            if (lane < group_size) {
                auto const first_reference = word * warp_width;
                auto const left = reference_count - first_reference;
                bits = left >= warp_width ? ~0U : (1U << left) - 1U;
                if constexpr (UpperTriangle) {
                    auto const query_id = query_id_offset + first_query + lane;
                    if (first_reference <= query_id) {
                        auto const skipped = query_id - first_reference + 1U;
                        bits &= skipped >= warp_width ? 0U : ~0U << skipped;
                    }
                }
            }
            uint32_t passed = 0U;
            auto remaining = __reduce_or_sync(0xffffffffU, bits);
            constexpr uint32_t refs = 4U;
            constexpr uint32_t bound_chunks = match_filter_planes / 4U;
            constexpr uint32_t bound_planes = match_filter_planes;
            while (remaining != 0U) {
                uint32_t bit[refs];
                bool has[refs];
                uint32_t last_bit = 0U;
                _Pragma("unroll")
                for (uint32_t i = 0; i < refs; ++i) {
                    has[i] = remaining != 0U;
                    bit[i] = has[i] ? static_cast<uint32_t>(__ffs(remaining) - 1) : bit[0];
                    last_bit = has[i] ? bit[i] : last_bit;
                    remaining &= remaining - 1U;
                }
                uint4 const* reference[refs];
                uint32_t selecting[refs];
                uint32_t r[refs][lane_groups][bound_planes];
                _Pragma("unroll")
                for (uint32_t i = 0; i < refs; ++i) {
                    reference[i] = reinterpret_cast<uint4 const*>(
                        reference_planes +
                        static_cast<size_t>(word * warp_width + bit[i]) * row_words
                    );
                    selecting[i] = has[i] ? __ballot_sync(0xffffffffU, (bits >> bit[i]) & 1U) : 0U;
                    _Pragma("unroll")
                    for (uint32_t j = 0; j < lane_groups; ++j) {
                        _Pragma("unroll")
                        for (uint32_t c = 0; c < bound_chunks; ++c) {
                            uint32_t words[4];
                            load_plane_chunk(reference[i], j * warp_width + lane, c, words);
                            _Pragma("unroll")
                            for (uint32_t w = 0; w < 4U; ++w) {
                                r[i][j][4U * c + w] = words[w];
                            }
                        }
                    }
                }
                // Selections are prefixes of the query group, growing with the reference ID.
                auto const members = static_cast<uint32_t>(
                    __popc(__ballot_sync(0xffffffffU, (bits >> last_bit) & 1U))
                );
                for (uint32_t m = 0U; m < members; ++m) {
                    auto const* member =
                        filter_query_planes + static_cast<size_t>(m) * (row_words / 4U);
                    uint32_t mismatches[refs] = {};
                    _Pragma("unroll")
                    for (uint32_t j = 0; j < lane_groups; ++j) {
                        uint32_t differ[refs];
                        _Pragma("unroll")
                        for (uint32_t i = 0; i < refs; ++i) {
                            differ[i] = excluded_lanes[j];
                        }
                        _Pragma("unroll")
                        for (uint32_t c = 0; c < bound_chunks; ++c) {
                            uint32_t q[4];
                            load_plane_chunk(member, j * warp_width + lane, c, q);
                            _Pragma("unroll")
                            for (uint32_t i = 0; i < refs; ++i) {
                                _Pragma("unroll")
                                for (uint32_t w = 0; w < 4U; ++w) {
                                    differ[i] |= q[w] ^ r[i][j][4U * c + w];
                                }
                            }
                        }
                        _Pragma("unroll")
                        for (uint32_t i = 0; i < refs; ++i) {
                            mismatches[i] += static_cast<uint32_t>(__popc(differ[i]));
                        }
                    }
                    _Pragma("unroll")
                    for (uint32_t i = 0; i < refs; ++i) {
                        auto const total_bound = static_cast<uint32_t>(BucketCount) -
                                                 __reduce_add_sync(0xffffffffU, mismatches[i]);
                        if (lane == m && ((selecting[i] >> m) & 1U) != 0U &&
                            total_bound >= minimum_matches) {
                            passed |= 1U << bit[i];
                        }
                    }
                }
            }
            if (lane < group_size) {
                candidate_bits[static_cast<size_t>(first_query + lane) * words_per_query + word] =
                    passed;
            }
        }
    }
}

Here is the call graph for this function:

mean_m()

__host__ __device__ double cuddl::detail::mean_m ( double  bucket_count, double  filled, double  sum_restored  )

inlinenoexcept

MeanM estimate retained as the full-sketch form of minimum_mle.

Definition at line 45 of file cardinality.cuh.

                                                                         {
    if (filled <= 0.0 || sum_restored <= 0.0) {
        return 0.0;
    }
    return bucket_count * filled * hash_range / sum_restored;
}

mean_m_f32()

__host__ __device__ float cuddl::detail::mean_m_f32 ( float  bucket_count, float  filled, float  sum_restored  )

inlinenoexcept

Single-precision mean_m for the throughput-bound cardinality reduction.

Definition at line 82 of file cardinality.cuh.

                                                                          {
    if (filled <= 0.0f || sum_restored <= 0.0f) {
        return 0.0f;
    }
    return bucket_count * filled * static_cast<float>(hash_range) / sum_restored;
}

minimum_mle()

__host__ __device__ double cuddl::detail::minimum_mle ( double  bucket_count, double  empty_count, double  sum_restored  )

inlinenoexcept

Estimates the number of distinct hashes from the minimum hash in each bucket.

Each distinct hash lands in one of bucket_count buckets. A bucket therefore receives about cardinality / bucket_count hashes. More hashes make its minimum smaller, so the observed minima tell us how many hashes must have produced them.

The denominator combines all observations after scaling hash values to [0, 1]:

• each empty bucket contributes 1, meaning that its minimum lies beyond the hash range;
• each filled bucket contributes its restored interval midpoint divided by hash_range.

Dividing bucket_count * filled_bucket_count by that total gives the maximum-likelihood cardinality estimate. If no bucket is empty, this reduces to the usual MeanM estimate.

Parameters

bucket_count	Total number of sketch buckets.
empty_count	Number of buckets that received no hashes.
sum_restored	Sum of the restored interval midpoint from every filled bucket.

Returns

Estimated distinct hash count, or zero when every bucket is empty.

Definition at line 33 of file cardinality.cuh.

                                                                                   {
    auto const filled = bucket_count - empty_count;
    if (filled <= 0.0) {
        return 0.0;
    }
    return bucket_count * filled / (empty_count + sum_restored / hash_range);
}

Here is the caller graph for this function:

minimum_mle_f32()

__host__ __device__ float cuddl::detail::minimum_mle_f32 ( float  bucket_count, float  empty_count, float  sum_restored  )

inlinenoexcept

Single-precision variants of the cardinality estimators.

The reduction dominates the cardinality kernel. Keeping the restored-sum reduction and the scalar estimate in FP32 avoids the 1/64 FP32:FP64 throughput ratio on consumer Blackwell while preserving far more precision than the ~2% standard error of a 2,048-register sketch.

Definition at line 71 of file cardinality.cuh.

                                                                                    {
    auto const filled = bucket_count - empty_count;
    if (filled <= 0.0f) {
        return 0.0f;
    }
    return bucket_count * filled /
           (empty_count + sum_restored / static_cast<float>(hash_range));
}

Here is the caller graph for this function:

mpi_try()

Result< void > cuddl::detail::mpi_try ( int  status, char const *  call  )

inline

Definition at line 65 of file distributed.cuh.

                                                                        {
    if (status == MPI_SUCCESS) return Ok();
    char text[MPI_MAX_ERROR_STRING];
    int length = 0;
    MPI_Error_string(status, text, &length);
    return Err(Error::resource(std::string(call) + ": " + std::string(text, length)));
}

Here is the call graph for this function:

Here is the caller graph for this function:

next_cell()

__device__ __forceinline__ uint32_t cuddl::detail::next_cell	(	uint32_t *	cell_counter,
		uint32_t	cell,
		uint32_t &	claimed
	)

The cell a refinement-style block processes after cell: claimed from cell_counter, a zeroed per-launch counter, so blocks balance uneven cells; without one, grid-stride.

Called by every thread of the block; claimed is the block's shared slot. The first cell is next_cell(cell_counter, blockIdx.x - gridDim.x, claimed).

Definition at line 1690 of file kernels.cuh.

                                                                    {
    if (cell_counter == nullptr) {
        return cell + gridDim.x;
    }
    __syncthreads();
    if (threadIdx.x == 0U) {
        claimed = atomicAdd(cell_counter, 1U);
    }
    __syncthreads();
    return claimed;
}

Here is the call graph for this function:

Here is the caller graph for this function:

pack_bases16_portable()

uint32_t cuddl::detail::pack_bases16_portable ( char const *  bases, size_t  count, uint32_t &  word  )

inline

Packs up to sixteen bases into a two-bit word, base j in bits 2j and 2j+1.

Codes are those of encode_base. A byte that is not a base gets a meaningless code.

Returns

A mask with bit j set when byte j is not a base, or is past count.

Definition at line 591 of file fastx_sequence_file.hpp.

                                                                                                     {
    if (count < 16) {
        uint32_t bad = 0;
        word = 0;
        for (uint32_t j = 0; j < 16; ++j) {
            auto const symbol = j < count ? encode_base(bases[j]) : uint8_t{0xFF};
            word |= static_cast<uint32_t>(symbol & 3U) << (2 * j);
            bad |= static_cast<uint32_t>(symbol == 0xFFU) << j;
        }
        return bad;
    }
    uint32_t bad = 0;
    word = 0;
    for (uint32_t half = 0; half < 2; ++half) {
        uint64_t x;
        std::memcpy(&x, bases + 8 * half, sizeof(x));
        constexpr uint64_t low7 = 0x7F7F7F7F7F7F7F7FULL;
        // High bit of each byte of v that is zero, without carries between bytes.
        auto const zero = [](uint64_t v) { return ~(((v & low7) + low7) | v | low7); };
        auto const upper = x & 0xDFDFDFDFDFDFDFDFULL;
        auto const valid = zero(upper ^ 0x4141414141414141ULL) |
                           zero(upper ^ 0x4343434343434343ULL) |
                           zero(upper ^ 0x4747474747474747ULL) | zero(upper ^ 0x5454545454545454ULL);
        auto codes = (x >> 1) & 0x0303030303030303ULL;
        codes = (codes | (codes >> 6)) & 0x000F000F000F000FULL;
        codes = (codes | (codes >> 12)) & 0x000000FF000000FFULL;
        codes = (codes | (codes >> 24)) & 0xFFFFULL;
        word |= static_cast<uint32_t>(codes) << (16 * half);
        // Byte j's flag lands on bit j.
        bad |= static_cast<uint32_t>(
                   ((~valid & 0x8080808080808080ULL) >> 7) * 0x0102040810204080ULL >> 56
               )
               << (8 * half);
    }
    return bad;
}

Here is the call graph for this function:

Here is the caller graph for this function:

pack_bases_pieces()

uint64_t cuddl::detail::pack_bases_pieces ( char const *  bases, size_t  count, uint32_t *  words, size_t  width  )

inline

Packs width bases from sixteen-base pieces, for a tail shorter than a full step.

Definition at line 629 of file fastx_sequence_file.hpp.

                                                                                                  {
    uint64_t bad = 0;
    for (size_t piece = 0; piece < width / 16; ++piece) {
        auto const left = count > 16 * piece ? count - 16 * piece : 0;
        bad |= uint64_t{pack_bases16_portable(bases + 16 * piece, left, words[piece])} << (16 * piece);
    }
    return bad;
}

Here is the call graph for this function:

pack_bases_portable()

uint64_t cuddl::detail::pack_bases_portable ( char const *  bases, size_t  count, uint32_t *  words  )

inline

Definition at line 638 of file fastx_sequence_file.hpp.

                                                                                      {
    return pack_bases16_portable(bases, count, words[0]);
}

Here is the call graph for this function:

Here is the caller graph for this function:

pack_fasta_chunks()

Result< void > cuddl::detail::pack_fasta_chunks ( fastx_sequence_file &  result, std::string_view  data, char *  out, uint32_t  k, std::string const &  path  )

inline

Compact and pack one cache-sized block at a time, without a genome-sized ASCII write.

Blocks end at line boundaries so header detection keeps its line-start invariant. Carrying k - 1 compacted bytes preserves every window without duplicating one.

Definition at line 1508 of file fastx_sequence_file.hpp.

  {
    size_t header = data.find('>');
    while (header != std::string_view::npos && header != 0 && data[header - 1] != '\n' &&
           data[header - 1] != '\r') {
        header = data.find('>', header + 1);
    }
    if (header == std::string_view::npos) {
        if (!data.empty()) return Err(Error::invalid_argument("FASTX parse error near: " + path));
        return Ok();
    }
    std::string scratch, carry;
    fastx_sequence_file chunk;
    uint64_t written = 0;
    bool has_sequence = false;
    auto const* const last = data.data() + data.size();
    while (true) {
        auto const end = fastx_line_end(data, header);
        if (end + 1 >= data.size()) break;
        auto const* at = data.data() + end + 1;
        carry.clear();
        while (at != last) {
            auto const offset = static_cast<size_t>(at - data.data());
            auto const boundary = offset + std::min<size_t>(16384, data.size() - offset);
            auto const limit = boundary == data.size()
                                   ? boundary
                                   : std::min(data.size(), fastx_line_end(data, boundary) + 1);
            auto const* const chunk_end = data.data() + limit;
            scratch.resize(carry.size() + static_cast<size_t>(chunk_end - at));
            std::memcpy(scratch.data(), carry.data(), carry.size());
            char const* stop = nullptr;
            auto* compact_end =
                fasta_sequence_compact(at, chunk_end, scratch.data() + carry.size(), stop);
            has_sequence |= stop != at;
            auto const bases = static_cast<size_t>(compact_end - scratch.data());
            carry.assign(compact_end - std::min<size_t>(k - 1, bases), compact_end);
            chunk.extents.push_back({scratch.data(), compact_end});
            pack_fastx_sequence_file(chunk, k);
            uint64_t words = 0;
            for (auto record : chunk.packed) {
                words = record.word + (record.bases + 15) / 16;
                record.word += written;
                result.packed.push_back(record);
            }
            std::memcpy(out + written * sizeof(uint32_t), scratch.data(), words * sizeof(uint32_t));
            written += words;
            chunk.packed.clear();
            at = stop;
            if (stop != chunk_end || stop == last) break;
        }
        if (at == last) break;
        header = static_cast<size_t>(at - data.data());
    }
    if (!has_sequence && !data.empty()) {
        return Err(Error::invalid_argument("FASTX parse error near: " + path));
    }
    result.words = reinterpret_cast<uint32_t const*>(out);
    result.input = {};
    return Ok();
}

Here is the call graph for this function:

Here is the caller graph for this function:

pack_fastx_sequence_file()

void cuddl::detail::pack_fastx_sequence_file ( fastx_sequence_file &  result, uint32_t  k  )

inline

Packs a parsed file's records in place into two-bit words, run by run.

A record splits at every byte that is not a base, and runs shorter than k are dropped: no k-mer holding such a byte is hashed, so the k-mers are those of the records. Output never passes unread input: words are written only after the packer has read their bytes, a full step writes a quarter of what it read, and a partial step writes only for a run of at least k >= 16 bases, whose words take at most half of the bytes it read.

Definition at line 761 of file fastx_sequence_file.hpp.

                                                                              {
    if (result.extents.empty() || k < 16) return;
    auto* const out = const_cast<char*>(result.extents.front().begin);
    uint64_t written = 0;
    for (auto const& extent : result.extents) {
        auto const* next = extent.begin;
        auto const* const end = extent.end;
        while (next < end) {
            while (next < end && encode_base(*next) == 0xFF) ++next;
            auto const* const start = next;
            auto const first = written;
            while (next < end) {
                uint32_t words[4];
                auto const bad = packer.pack(next, static_cast<size_t>(end - next), words);
                auto const bases =
                    bad == 0 ? packer.width : static_cast<size_t>(std::countr_zero(bad));
                // A partial step ends the run; a short run's partial words are never written.
                if (bases == packer.width || static_cast<size_t>(next + bases - start) >= k) {
                    auto const count = (bases + 15) / 16;
                    std::memcpy(out + written * sizeof(uint32_t), words, count * sizeof(uint32_t));
                    written += count;
                }
                next += bases;
                if (bases != packer.width) break;
            }
            auto const bases = static_cast<uint64_t>(next - start);
            if (bases >= k) {
                result.packed.push_back({first, bases});
            } else {
                written = first;
            }
        }
    }
    result.words = reinterpret_cast<uint32_t const*>(out);
    result.extents.clear();
}

Here is the call graph for this function:

Here is the caller graph for this function:

pages_locked()

bool cuddl::detail::pages_locked ( transfer_mode  mode, cuda::device_ref  device  )

inlinenoexcept

Whether a build in mode decompresses into page-locked buffers.

automatic keys on the device rather than the architecture: a device that reads pageable host memory is coherent, and there the host writes to page-locked memory are the slow part.

Definition at line 99 of file database_staging.cuh.

                                                                                           {
    if (mode == transfer_mode::pinned) return true;
    if (mode != transfer_mode::automatic) return false;
    return !device_reads_pageable_memory(device);
}

Here is the call graph for this function:

Here is the caller graph for this function:

parallel_for()

template<typename Work >

void cuddl::detail::parallel_for	(	size_t	count,
		size_t	workers,
		Work &&	work
	)

Runs work(index) for every index below count on workers threads.

Definition at line 203 of file device_gzip.cuh.

                                                             {
    std::atomic<size_t> next{0};
    auto run = [&] {
        for (size_t index = next++; index < count; index = next++) work(index);
    };
    std::vector<std::jthread> threads;
    auto const helpers = std::min(workers, count) > 0 ? std::min(workers, count) - 1 : 0;
    threads.reserve(helpers);
    for (size_t thread = 0; thread < helpers; ++thread) threads.emplace_back(run);
    run();
}

parse_fasta()

Result< fasta_parse_result > cuddl::detail::parse_fasta ( std::string const &  path, uint32_t  k, unsigned  threads = std::thread::hardware_concurrency()  )

inline

Parses a FASTA/FASTQ file's sequence bases into packed canonical k-mers of length k.

Record headers and FASTQ qualities are excluded, and every record's sequence is fed through the DDL rolling-window extraction as one combined genome. An invalid or ambiguous base breaks the rolling window, and no k-mer spanning it is emitted. Each k-mer is canonicalised to the larger packed orientation. Files above 1 MiB are parsed in parallel: the sequence stream is split into seeded spans (fastx_split_sequence_spans), each worker replays its window prefix and then consumes its segments, and the per-span results concatenate to the byte-identical single-threaded output.

Parameters

path	FASTA/FASTQ file to parse.
k	K-mer length.
threads	Worker count for parallel files, defaulting to std::thread::hardware_concurrency(); files at or below 1 MiB parse serially regardless.

Returns

A fasta_parse_result, or an error if the file cannot be opened, mapped, or contains bytes but no FASTA records, or malformed FASTQ records.

Definition at line 288 of file fasta_parser.hpp.

  {
    auto source = CUDDL_TRY(load_fastx_sequence_file(path));
    auto& extents = source->extents;
    if (extents.empty()) return fasta_parse_result{};
 
    // The span splitter otherwise joins extents and seeds windows across record headers.
    // A separate invalid-base extent stops prefix replay. Workers recognize its address and
    // reset without counting the synthetic separator as an invalid input window.
    static constexpr char record_boundary = 'N';
    if (extents.size() > 1) {
        std::vector<fastx_sequence_extent> separated;
        separated.reserve(extents.size() * 2 - 1);
        for (auto const& extent : extents) {
            if (!separated.empty()) separated.push_back({&record_boundary, &record_boundary + 1});
            separated.push_back(extent);
        }
        extents = std::move(separated);
    }
 
    uint64_t total = 0;
    for (auto const& extent : extents) {
        total += static_cast<uint64_t>(extent.end - extent.begin);
    }
 
    unsigned worker_count = 1;
    if (total >= (uint64_t{1} << 20)) {
        auto const available = threads > 0 ? threads : 1U;
        worker_count = available > 0 ? available : 1U;
    }
 
    auto const spans = fastx_split_sequence_spans(extents, k > 0 ? k - 1U : 0U, worker_count);
 
    auto const mask = (k >= 32) ? ~uint64_t{0} : ((1ULL << (2U * k)) - 1ULL);
    std::vector<span_result> partials(spans.size());
    auto const per_thread = total / spans.size() + (total % spans.size() != 0U);
    for (auto& partial : partials) {
        partial.kmers.reserve(per_thread / 2U + 64U);
    }
 
    std::vector<std::thread> workers;
    workers.reserve(spans.size());
    for (size_t t = 0; t < spans.size(); ++t) {
        workers.emplace_back([&, t] {
            uint64_t window = 0;
            uint32_t window_len = 0;
            span_result sink{};  // prefix replay: no counts, no emissions
            for (char ch : spans[t].prefix) {
                consume_byte(ch, k, mask, sink, window, window_len);
            }
            for (auto const& segment : spans[t].segments) {
                if (segment.data() == &record_boundary) {
                    window = 0;
                    window_len = 0;
                    continue;
                }
                for (char ch : segment) {
                    consume_byte(ch, k, mask, partials[t], window, window_len);
                }
            }
        });
    }
    for (auto& worker : workers) {
        worker.join();
    }
 
    fasta_parse_result result;
    uint64_t total_valid = 0;
    for (auto const& partial : partials) {
        result.bases += partial.bases;
        total_valid += partial.valid_kmers;
        result.invalid_windows += partial.invalid_windows;
    }
    result.valid_kmers = total_valid;
    // One contiguous allocation for the final stream: reserve the exact count, then move each
    // partial in and release it immediately so the peak never exceeds the final size plus one
    // partial (the machine's working set is the result stream itself).
    result.kmers.reserve(total_valid);
    for (auto& partial : partials) {
        result.kmers.insert(
            result.kmers.end(),
            std::make_move_iterator(partial.kmers.begin()),
            std::make_move_iterator(partial.kmers.end())
        );
        partial.kmers = std::vector<uint64_t>{};
    }
    return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

parse_fastx_sequence_file()

Result< void > cuddl::detail::parse_fastx_sequence_file ( fastx_sequence_file &  result, std::string const &  path  )

inline

Definition at line 1470 of file fastx_sequence_file.hpp.

                                                                              {
    auto const data = std::exchange(result.input, {});
    auto& sequence = result.sequence;
    auto& extents = result.extents;
    auto const first = data.find_first_not_of("\r\n");
    if (first != std::string_view::npos && data[first] == '@') {
        fastx_fastq_reader reader{data, path};
        std::vector<std::pair<size_t, size_t>> records;
        while (true) {
            auto const begin = sequence.size();
            auto next = reader.append_next_record(sequence);
            if (!next) return Err(Error::invalid_argument(next.error().message()));
            if (!*next) break;
            if (sequence.size() > begin) records.emplace_back(begin, sequence.size());
        }
        if (sequence.empty()) return Ok();
        for (auto const& [begin, end] : records) {
            extents.push_back({sequence.data() + begin, sequence.data() + end});
        }
        compact_fastx_sequence_extents(result);
        return Ok();
    }
    // Mapped plain bytes are read-only; decompressed bytes can be compacted in place.
    char* out = result.decompressed_target;
    if (out == nullptr) {
        if (data.data() != result.decompressed.data()) result.decompressed.resize(data.size());
        out = result.decompressed.data();
    }
    extents = fastx_fasta_compact_extents(data, out);
    if (extents.empty() && !data.empty()) {
        return Err(Error::invalid_argument("FASTX parse error near: " + path));
    }
    return Ok();
}

Here is the call graph for this function:

Here is the caller graph for this function:

plan_staging()

template<uint32_t K, size_t BucketCount>

Result< staging_plan > cuddl::detail::plan_staging ( size_t  references, uint64_t  staged_ceiling, std::optional< size_t >  staging_bytes, cuda::stream_ref  stream  )

inline

Sizes the arena and the row store from free device memory and the corpus.

staged_ceiling bounds the bytes the corpus can stage, which keeps a small collection from reserving an arena it could never fill. An unset staging_bytes sizes the arena from what is free; a value is honoured, or rejected when it does not fit.

Definition at line 143 of file database_staging.cuh.

  {
    size_t const available = CUDDL_TRY(available_device_bytes(stream));
    size_t const usable = available - available / 10;
    size_t const row_bytes = BucketCount * sizeof(uint32_t);
    // Rows are small next to the input, so the whole collection stays resident whenever that
    // costs at most a quarter of what is affordable; otherwise rows stream back one group at a
    // time as each group completes.
    size_t const group = std::min(references, std::max<size_t>(1, usable / (4 * row_bytes)));
    // Four fifths of what is affordable goes to the arena, which leaves room for the row store,
    // the descriptors and whatever the caller keeps on the device. On a coherent CPU/GPU system
    // the free memory reported here is host memory, so a share of host memory bounds it as well.
    auto const pages = ::sysconf(_SC_PHYS_PAGES);
    auto const page = ::sysconf(_SC_PAGE_SIZE);
    size_t const host_share = pages > 0 && page > 0
                                  ? static_cast<size_t>(pages) * static_cast<size_t>(page) / 8
                                  : std::numeric_limits<size_t>::max();
    size_t const arena_ceiling = static_cast<size_t>(std::min(
        {staged_ceiling,
         static_cast<uint64_t>(usable - usable / 5),
         static_cast<uint64_t>(host_share),
         static_cast<uint64_t>(default_arena_ceiling)}
    ));
    size_t const row_store_bytes = group * row_bytes;
    size_t const after_rows = usable > row_store_bytes ? usable - row_store_bytes : 0;
    if (staging_bytes.has_value() && *staging_bytes < K + 1) {
        return Err(Error::invalid_argument("an explicit staging arena must hold one window"));
    }
    // Every arena gets the same share: one fills while kernels read the others.
    size_t const staging = staging_bytes.has_value()
                               ? *staging_bytes
                               : std::min(after_rows / stager_arenas, arena_ceiling);
    // One piece of at least k bases plus its k - 1 byte overlap has to fit.
    if (staging < K + 1 || staging > after_rows / stager_arenas) {
        return Err(
            Error::resource(
                "insufficient free device memory for reference staging (free " +
                std::to_string(available >> 20) + " MiB)"
            )
        );
    }
    // One descriptor per staged piece; the cap keeps a corpus of very short records from
    // demanding a descriptor per record.
    size_t const max_pieces = std::min<size_t>(size_t{1} << 20, std::max<size_t>(64, staging / K));
    // Pieces are a quarter of the arena at most, so a batch always holds several, and short
    // enough that a piece's window count stays representable.
    size_t const max_piece = std::min(
        {std::max<size_t>(K, staging / 4),
         static_cast<size_t>(std::numeric_limits<uint32_t>::max()) - K + 1}
    );
    return staging_plan{group, staging, max_piece, max_pieces};
}

Here is the call graph for this function:

plane_any()

__device__ __forceinline__ uint32_t cuddl::detail::plane_any ( uint32_t const (&)  planes[score_planes] )

noexcept

Buckets of a plane group holding a nonzero score.

Definition at line 1421 of file kernels.cuh.

                                                           {
    uint32_t any = 0U;
    _Pragma("unroll")
    for (uint32_t p = 0; p < score_planes; ++p) {
        any |= planes[p];
    }
    return any;
}

Here is the call graph for this function:

Here is the caller graph for this function:

plane_low_any()

__device__ __forceinline__ uint32_t cuddl::detail::plane_low_any ( uint32_t const (&)  planes[score_planes] )

noexcept

Buckets of a plane group whose score is nonzero below the top plane.

Definition at line 1432 of file kernels.cuh.

                                                               {
    uint32_t any = 0U;
    _Pragma("unroll")
    for (uint32_t p = 0; p + 1U < score_planes; ++p) {
        any |= planes[p];
    }
    return any;
}

Here is the call graph for this function:

probe_gzip_file()

gzip_file_probe cuddl::detail::probe_gzip_file ( std::filesystem::path const &  path )

inline

Reads the header and trailer of path, and of every block of a BGZF file.

A candidate is a gzip file without FEXTRA whose trailer names a non-empty member, a BGZF file with a non-empty block, or an uncompressed file that opens with a FASTA header, any of them with a name that does not mark it FASTQ. For gzip without FEXTRA, additional member signatures are checked after reading the compressed bytes, before accepting the device output.

Definition at line 135 of file device_gzip.cuh.

                                                                                    {
    gzip_file_probe probe;
    int const fd = ::open(path.c_str(), O_RDONLY);
    if (fd == -1) return probe;
    struct stat status{};
    unsigned char header[18] = {};
    unsigned char trailer[8 + 18] = {};  // a member's trailer, and the next BGZF header
    if (::fstat(fd, &status) == 0 && status.st_size >= 4 &&
        ::pread(fd, header, sizeof(header), 0) >= 4) {
        auto const size = static_cast<uint64_t>(status.st_size);
        probe.compressed = size;
        auto name = path.filename().string();
        std::transform(name.begin(), name.end(), name.begin(), [](unsigned char c) {
            return static_cast<char>(std::tolower(c));
        });
        auto const fastq =
            name.find(".fq") != std::string::npos || name.find(".fastq") != std::string::npos;
        if (header[0] == '>') {
            // Slot offsets count a plain file's bytes and its guard byte in 32 bits.
            probe.plain = size < std::numeric_limits<uint32_t>::max();
            probe.isize = static_cast<uint32_t>(probe.plain ? size : 0);
            probe.device = probe.plain && !fastq;
        } else if (size >= 26 && bgzf_block_size(header) != 0) {
            // One read per block: its trailer, and the header of the block after it.
            uint64_t at = 0, total = 0;
            uint32_t crc = 0, block_size = bgzf_block_size(header);
            bool tiled = false;
            while (block_size >= 26 && block_size <= size - at) {
                auto const next = at + block_size;
                auto const wanted = next == size ? 8 : std::min<uint64_t>(26, size - next + 8);
                if (::pread(fd, trailer, wanted, static_cast<off_t>(next - 8)) !=
                    static_cast<ssize_t>(wanted)) {
                    break;
                }
                uint32_t block_crc = 0, inflated = 0;
                std::memcpy(&block_crc, trailer, 4);
                std::memcpy(&inflated, trailer + 4, 4);
                if (inflated != 0) {
                    crc = crc32_combine(crc, block_crc, inflated);
                    total += inflated;
                    ++probe.members;
                }
                if (next == size) {
                    tiled = true;
                    break;
                }
                block_size = wanted == 26 ? bgzf_block_size(trailer + 8) : 0;
                at = next;
            }
            probe.bgzf = true;
            probe.crc = crc;
            probe.isize = static_cast<uint32_t>(total);
            probe.device =
                tiled && total != 0 && total < std::numeric_limits<uint32_t>::max() && !fastq;
        } else if (size >= 18 && ::pread(fd, trailer, 8, static_cast<off_t>(size - 8)) == 8) {
            std::memcpy(&probe.crc, trailer, 4);
            std::memcpy(&probe.isize, trailer + 4, 4);
            probe.members = 1;
            probe.device = header[0] == 0x1f && header[1] == 0x8b && header[2] == 8 &&
                           (header[3] & 0xe4U) == 0 && probe.isize != 0 && !fastq;
        }
    }
    ::close(fd);
    return probe;
}

Here is the call graph for this function:

propagate_error()

cuda::std::unexpected< Error > cuddl::detail::propagate_error ( const Error &  error )

inline

Copies error for propagation (avoids moving out of expected::error()).

Definition at line 169 of file error.hpp.

                                                                                  {
    return cuda::std::unexpected<Error>(Error{error});
}

reference_database_digest()

Result< uint64_t > cuddl::detail::reference_database_digest ( reference_database_file const &  db )

inline

Definition at line 42 of file reference_index_file.cuh.

                                                                                     {
    reference_index_digest hash;
    CUDDL_TRY(hash.bytes("CUDDLDB-XXH64-v1", sizeof("CUDDLDB-XXH64-v1") - 1));
    auto const metadata = db.metadata();
    auto const& c = metadata.compatibility;
    CUDDL_TRY(hash.value(c.kmer_length));
    CUDDL_TRY(hash.value(c.bucket_count));
    CUDDL_TRY(hash.value(c.indexed_bucket_count));
    CUDDL_TRY(hash.value(c.score_encoder_identity));
    CUDDL_TRY(hash.value(c.exponent_bits));
    CUDDL_TRY(hash.value(c.mantissa_bits));
    CUDDL_TRY(hash.value(c.hash_identity));
    CUDDL_TRY(hash.value(c.hash_seed));
    CUDDL_TRY(hash.value(c.canonicalisation_policy));
    CUDDL_TRY(hash.value(c.blacklist_identity));
    CUDDL_TRY(hash.value(c.blacklist_version));
    CUDDL_TRY(hash.value(c.key_mask));
    CUDDL_TRY(hash.value(metadata.reference_count));
    CUDDL_TRY(hash.value(static_cast<uint64_t>(db.names().size())));
    for (auto const& name : db.names()) {
        CUDDL_TRY(hash.value(static_cast<uint64_t>(name.size())));
        CUDDL_TRY(hash.bytes(name.data(), name.size()));
    }
    CUDDL_TRY(hash.words(db.rows()));
    return hash.digest;
}

Here is the call graph for this function:

Here is the caller graph for this function:

refine_all_pairs_kernel()

template<size_t BucketCount, typename SearchResult , bool UpperTriangle, uint32_t QueryGroup>

__global__ void cuddl::detail::refine_all_pairs_kernel	(	uint32_t const *	query_planes,
		uint32_t	query_id_offset,
		uint32_t	query_count,
		uint32_t const *	reference_planes,
		uint32_t	reference_count,
		SearchResult *	results,
		uint32_t *	cell_counter,
		uint32_t const *	reference_mask
	)

Exactly compares the layout pairs selected by reference_mask, without match counts.

A null mask selects every reference.

Cells and query staging follow refine_batch_bitmap_kernel; warps claim all_pairs_word_parts parts of a 32-reference word at a time. A warp takes a claim's references two at a time and keeps both references' planes in registers, so each query plane group read from shared memory serves two pairs, and the two comparisons give every warp independent logic chains to interleave.

Definition at line 2316 of file kernels.cuh.

  {
    static_assert(match_filter_supported<BucketCount>);
    constexpr uint32_t warp_width = 32;
    constexpr uint32_t group = QueryGroup;
    static_assert(group >= 1U && group <= warp_width);
    constexpr uint32_t row_words = static_cast<uint32_t>(BucketCount / 2U);
    constexpr uint32_t lane_groups = static_cast<uint32_t>(BucketCount / warp_width / warp_width);
    constexpr uint32_t max_block_words = static_cast<uint32_t>(cuda::std::max<size_t>(
        1U, refine_block_row_bytes / (BucketCount * sizeof(uint16_t)) / warp_width
    ));
    extern __shared__ uint4 all_pairs_query_planes[];
    __shared__ uint32_t next_word;
    constexpr uint32_t part_width = warp_width / all_pairs_word_parts;
    __shared__ uint32_t claimed_cell;
 
    auto const lane = static_cast<uint32_t>(threadIdx.x) % warp_width;
    auto const words_per_query = candidate_bit_words(reference_count);
    auto const cells = make_refine_cells<UpperTriangle>(
        query_id_offset, query_count, group, words_per_query, max_block_words, 1U
    );
    auto const groups = cells.groups;
 
    for (auto cell = next_cell(cell_counter, blockIdx.x - gridDim.x, claimed_cell);
         cell < cells.cell_count;
         cell = next_cell(cell_counter, cell, claimed_cell)) {
        auto const first_query = (cell % groups) * group;
        auto const group_size = cuda::std::min(group, query_count - first_query);
        auto const first_word = cells.first_word + (cell / groups) * cells.block_words;
        auto const last_word = cuda::std::min(first_word + cells.block_words, words_per_query);
        auto const* group_planes = reinterpret_cast<uint4 const*>(
            query_planes + static_cast<size_t>(first_query) * row_words
        );
 
        __syncthreads();
        if (threadIdx.x == 0U) {
            next_word = first_word * all_pairs_word_parts;
        }
        for (auto i = static_cast<uint32_t>(threadIdx.x); i < group_size * (row_words / 4U);
             i += all_pairs_block_size) {
            all_pairs_query_planes[i] = group_planes[i];
        }
        __syncthreads();
 
        for (;;) {
            uint32_t word = 0U;
            if (lane == 0U) {
                word = atomicAdd(&next_word, 1U);
            }
            word = __shfl_sync(0xffffffffU, word, 0);
            auto const part = word % all_pairs_word_parts;
            word /= all_pairs_word_parts;
            if (word >= last_word) {
                break;
            }
            // Lane m holds query m's selected references in this part of the word.
            uint32_t bits = 0U;
            if (lane < group_size) {
                auto const first_reference = word * warp_width;
                auto const left = reference_count - first_reference;
                bits = left >= warp_width ? ~0U : (1U << left) - 1U;
                bits &= ((1U << part_width) - 1U) << (part * part_width);
                if constexpr (UpperTriangle) {
                    auto const query_id = query_id_offset + first_query + lane;
                    if (first_reference <= query_id) {
                        auto const skipped = query_id - first_reference + 1U;
                        bits &= skipped >= warp_width ? 0U : ~0U << skipped;
                    }
                }
                if (reference_mask != nullptr) {
                    bits &= reference_mask[word];
                }
            }
            auto remaining = __reduce_or_sync(0xffffffffU, bits);
            while (remaining != 0U) {
                auto const bit_a = static_cast<uint32_t>(__ffs(remaining) - 1);
                remaining &= remaining - 1U;
                auto const has_b = remaining != 0U;
                auto const bit_b = has_b ? static_cast<uint32_t>(__ffs(remaining) - 1) : bit_a;
                remaining &= remaining - 1U;
                auto const reference_a = word * warp_width + bit_a;
                auto const reference_b = word * warp_width + bit_b;
                auto const* planes_a = reinterpret_cast<uint4 const*>(
                    reference_planes + static_cast<size_t>(reference_a) * row_words
                );
                auto const* planes_b = reinterpret_cast<uint4 const*>(
                    reference_planes + static_cast<size_t>(reference_b) * row_words
                );
                uint32_t a[lane_groups][score_planes];
                uint32_t b[lane_groups][score_planes];
                uint32_t a_any[lane_groups];
                uint32_t b_any[lane_groups];
                uint32_t nonzero = ~0U;
                _Pragma("unroll")
                for (uint32_t j = 0; j < lane_groups; ++j) {
                    load_plane_group(planes_a, j * warp_width + lane, a[j]);
                    load_plane_group(planes_b, j * warp_width + lane, b[j]);
                    a_any[j] = plane_any(a[j]);
                    b_any[j] = plane_any(b[j]);
                    nonzero &= a_any[j] & b_any[j];
                }
                // Selections are prefixes of the query group, and B's includes A's.
                auto const selected_a = __ballot_sync(0xffffffffU, (bits >> bit_a) & 1U);
                auto const selected_b = __ballot_sync(0xffffffffU, (bits >> bit_b) & 1U);
                auto const members = static_cast<uint32_t>(__popc(selected_b));
                // Lane m keeps member m's totals for both references.
                pairwise_counts mine_a{};
                pairwise_counts mine_b{};
                auto const compare = [&](auto full) {
                    for (uint32_t m = 0U; m < members; ++m) {
                        auto const* member =
                            all_pairs_query_planes + static_cast<size_t>(m) * (row_words / 4U);
                        plane_counts counts_a{};
                        plane_counts counts_b{};
                        _Pragma("unroll")
                        for (uint32_t j = 0; j < lane_groups; ++j) {
                            uint32_t q[score_planes];
                            load_plane_group(member, j * warp_width + lane, q);
                            compare_plane_group<false, decltype(full)::value>(
                                q, a[j], a_any[j], 0U, 0U, 0U, counts_a
                            );
                            compare_plane_group<false, decltype(full)::value>(
                                q, b[j], b_any[j], 0U, 0U, 0U, counts_b
                            );
                        }
                        auto const total_a = counts_a.reduce<BucketCount, decltype(full)::value>();
                        auto const total_b = counts_b.reduce<BucketCount, decltype(full)::value>();
                        if (lane == m) {
                            mine_a = total_a;
                            mine_b = total_b;
                        }
                    }
                };
                if (__all_sync(0xffffffffU, nonzero == ~0U)) {
                    compare(cuda::std::true_type{});
                } else {
                    compare(cuda::std::false_type{});
                }
                // The slot implies the pair; only its counts are stored.
                if (((selected_a >> lane) & 1U) != 0U) {
                    results[batch_result_slot<UpperTriangle>(
                        first_query + lane, reference_a, query_id_offset, reference_count
                    )] = SearchResult::pack(mine_a.lower, mine_a.equal, mine_a.higher);
                }
                if (has_b && ((selected_b >> lane) & 1U) != 0U) {
                    results[batch_result_slot<UpperTriangle>(
                        first_query + lane, reference_b, query_id_offset, reference_count
                    )] = SearchResult::pack(mine_b.lower, mine_b.equal, mine_b.higher);
                }
            }
        }
    }
}

Here is the call graph for this function:

refine_write_result()

template<typename SearchResult >

__device__ void cuddl::detail::refine_write_result	(	uint32_t	index,
		uint32_t	query_index,
		uint32_t	reference_id,
		uint32_t	query_id_offset,
		uint32_t const *	match_counts,
		uint32_t	pair_id,
		pairwise_counts	local,
		uint32_t	lane,
		SearchResult *	results,
		uint32_t *	result_match_counts
	)

Warp-reduces one refined candidate and writes its stable result slot.

Each counter reduces with one redux.sync instruction instead of a shuffle tree.

Definition at line 1290 of file kernels.cuh.

  {
    pairwise_counts const total{
        .lower = __reduce_add_sync(0xffffffffU, local.lower),
        .equal = __reduce_add_sync(0xffffffffU, local.equal),
        .higher = __reduce_add_sync(0xffffffffU, local.higher),
        .both_empty = __reduce_add_sync(0xffffffffU, local.both_empty),
    };
    if (lane == 0U) {
        results[index].query_id = query_id_offset + query_index;
        results[index].reference_id = reference_id;
        results[index].counts = total;
        if (result_match_counts != nullptr) {
            result_match_counts[index] = match_counts[pair_id];
        }
    }
}

resolve_bases_packer()

bases_packer cuddl::detail::resolve_bases_packer ( )

inline

Selects the widest base packer the host supports.

Definition at line 739 of file fastx_sequence_file.hpp.

                                                         {
    if (ablated("simd-parsing")) return {pack_bases_portable, 16};
#if (defined(__x86_64__) || defined(__i386__)) && (defined(__GNUC__) || defined(__clang__))
    if (__builtin_cpu_supports("avx512bw")) return {pack_bases_avx512, 64};
    if (__builtin_cpu_supports("avx2")) return {pack_bases_avx2, 32};
#endif
#if defined(__aarch64__) || defined(_M_ARM64)
    return {pack_bases_neon, 64};
#else
    return {pack_bases_portable, 16};
#endif
}

Here is the call graph for this function:

resolve_fastx_line_end()

fastx_line_end_fn cuddl::detail::resolve_fastx_line_end ( )

inline

Selects the fastest line-end scanner the host supports.

Definition at line 178 of file fastx_sequence_file.hpp.

                                                                {
    if (ablated("simd-parsing")) return fastx_line_end_scalar;
#if defined(__ARM_FEATURE_SVE)
    return fastx_line_end_sve;
#elif defined(__aarch64__) || defined(_M_ARM64)
    return fastx_line_end_neon;
#elif (defined(__GNUC__) || defined(__clang__)) && defined(__x86_64__)
    if (__builtin_cpu_supports("avx512bw")) return fastx_line_end_avx512;
    return __builtin_cpu_supports("avx2") ? fastx_line_end_avx2 : fastx_line_end_scalar;
#else
    return fastx_line_end_scalar;
#endif
}

Here is the call graph for this function:

resolve_gzip_signature_scanner()

auto cuddl::detail::resolve_gzip_signature_scanner ( ) -> bool (*)(char const*, size_t)

inline

Selects the widest signature scanner the host supports.

Definition at line 305 of file device_gzip.cuh.

                                                                                            {
    if (ablated("simd-parsing")) return has_gzip_signature_portable;
#if (defined(__x86_64__) || defined(__i386__)) && (defined(__GNUC__) || defined(__clang__))
    if (__builtin_cpu_supports("avx512bw")) return has_gzip_signature_avx512;
    if (__builtin_cpu_supports("avx2")) return has_gzip_signature_avx2;
#endif
#if defined(__aarch64__) || defined(_M_ARM64)
    return has_gzip_signature_neon;
#endif
    return has_gzip_signature_portable;
}

Here is the call graph for this function:

restore()

template<typename Layout = default_register_layout>

__host__ __device__ constexpr uint64_t cuddl::detail::restore ( uint16_t  stored )

constexprnoexcept

Reconstructs the approximate original hash magnitude from a stored score.

Inverts the score encoding: recovers NLZ from the exponent field, uninverts the mantissa bits and prepends the implicit leading one, then shifts the result back into 64-bit space.

Definition at line 68 of file register.cuh.

                                                                         {
    constexpr auto layout_mantissa_mask = (1U << Layout::mantissa_bits) - 1U;
    constexpr auto layout_restore_shift_base = 63U - Layout::mantissa_bits;
    auto const nlz = static_cast<uint32_t>(stored >> Layout::mantissa_bits);
    auto const lowbits = static_cast<uint32_t>((~stored) & layout_mantissa_mask);
    auto const mantissa = (1U << Layout::mantissa_bits) | lowbits;
    auto const shift = layout_restore_shift_base - nlz;
    return static_cast<uint64_t>(mantissa) << shift;
}

restore_midpoint()

template<typename Layout = default_register_layout>

__host__ __device__ constexpr uint64_t cuddl::detail::restore_midpoint ( uint16_t  stored )

constexprnoexcept

Reconstructs the interval midpoint for restore.

restore returns the lower bound of the hash interval represented by stored. The actual hash is uniformly distributed across that interval, so the midpoint is the minimum-variance scalar summary and reduces the small positive bias that lower-bound restoration introduces into hash-magnitude cardinality sums.

Definition at line 87 of file register.cuh.

                                                                                  {
    constexpr auto layout_mantissa_mask = (1U << Layout::mantissa_bits) - 1U;
    constexpr auto layout_restore_shift_base = 63U - Layout::mantissa_bits;
    auto const nlz = static_cast<uint32_t>(stored >> Layout::mantissa_bits);
    auto const lowbits = static_cast<uint32_t>((~stored) & layout_mantissa_mask);
    auto const mantissa = (1U << Layout::mantissa_bits) | lowbits;
    auto const shift = layout_restore_shift_base - nlz;
    auto const lower = static_cast<uint64_t>(mantissa) << shift;
    // `(1 << shift) >> 1` is 2^(shift-1) for ordinary tiers and exactly zero for the
    // clamped top tier (`shift == 0`), so the midpoint needs no branch.
    return lower + ((1ULL << shift) >> 1U);
}

reverse_complement()

constexpr uint64_t cuddl::detail::reverse_complement ( uint64_t  packed, uint32_t  k  )

inlineconstexprnoexcept

Reverse-complements a packed k-mer (2 bits per base) with bit-parallel reversal.

Reverses the order of the 2-bit base digits with mask-swap rounds, shifts the reversed 2k bits down from the high end, then complements every base (^0b10). Byte-identical to the scalar loop formulation for every k <= 32; longer k-mers fall back to the scalar loop. The __host__ __device__ annotation only exists under the CUDA compiler so plain host translation units see a normal inline function, as with encode_base.

Definition at line 42 of file fasta_parser.hpp.

                                                                                   {
    if (k > 32) {
        uint64_t result = 0;
        for (uint32_t base = 0; base < k; ++base) {
            result = (result << 2U) | static_cast<uint64_t>(static_cast<uint8_t>(packed & 3U) ^ 2U);
            packed >>= 2U;
        }
        return result;
    }
    uint64_t x = packed;
    x = ((x & 0xCCCCCCCCCCCCCCCCULL) >> 2U) | ((x & 0x3333333333333333ULL) << 2U);
    x = ((x & 0xF0F0F0F0F0F0F0F0ULL) >> 4U) | ((x & 0x0F0F0F0F0F0F0F0FULL) << 4U);
    x = ((x & 0xFF00FF00FF00FF00ULL) >> 8U) | ((x & 0x00FF00FF00FF00FFULL) << 8U);
    x = ((x & 0xFFFF0000FFFF0000ULL) >> 16U) | ((x & 0x0000FFFF0000FFFFULL) << 16U);
    x = (x >> 32U) | (x << 32U);
    x >>= (64U - 2U * k);
    auto const mask = (k == 32) ? ~uint64_t{0} : ((1ULL << (2U * k)) - 1ULL);
    return (x ^ 0xAAAAAAAAAAAAAAAAULL) & mask;
}

Here is the caller graph for this function:

rotl64()

constexpr __host__ __device__ __forceinline__ uint64_t cuddl::detail::rotl64 ( uint64_t  x, int8_t  r  )

constexpr

Rotates x left by r bits.

Definition at line 40 of file hash.cuh.

                                                                                    {
    return cuda::std::rotl(x, static_cast<int>(r));
}

Here is the caller graph for this function:

row_type()

Result< MPI_Datatype > cuddl::detail::row_type ( uint32_t  buckets )

inline

One row of buckets scores as a single MPI element, so counts of rows stay within int.

Definition at line 264 of file distributed.cuh.

                                                                     {
    MPI_Datatype row = MPI_DATATYPE_NULL;
    CUDDL_TRY(mpi_try(
        MPI_Type_contiguous(static_cast<int>(buckets), MPI_UINT16_T, &row), "MPI_Type_contiguous"
    ));
    CUDDL_TRY(mpi_try(MPI_Type_commit(&row), "MPI_Type_commit"));
    return row;
}

Here is the call graph for this function:

Here is the caller graph for this function:

score()

template<typename Layout = default_register_layout>

__host__ __device__ uint16_t cuddl::detail::score ( uint64_t  hash )

inlinenoexcept

Computes the 16-bit DDL rarity score for hash.

The exponent field stores the number of leading zeros (NLZ); the mantissa field stores the mantissa_bits bits immediately following the leading one, bitwise inverted. A larger score always encodes a rarer hash. Zero is reserved for the empty register, so a hash whose raw exponent/mantissa encoding is zero collapses to one.

Definition at line 43 of file register.cuh.

                                                                  {
    constexpr auto layout_mantissa_mask = (1U << Layout::mantissa_bits) - 1U;
    constexpr auto layout_restore_shift_base = 63U - Layout::mantissa_bits;
    constexpr auto layout_max_nlz =
        cuda::std::min(layout_restore_shift_base, (1U << Layout::exponent_bits) - 1U);
#ifdef __CUDA_ARCH__
    auto const nlz = static_cast<uint32_t>(__clzll(hash | 1ULL));
#else
    auto const nlz = static_cast<uint32_t>(__builtin_clzll(hash | 1ULL));
#endif
    auto const capped = nlz > layout_max_nlz ? layout_max_nlz : nlz;
    auto const shift = layout_restore_shift_base - capped;
    auto const mantissa = static_cast<uint32_t>((hash >> shift) & layout_mantissa_mask);
    auto const inverted = static_cast<uint32_t>(mantissa ^ layout_mantissa_mask);
    auto const raw = static_cast<uint16_t>((capped << Layout::mantissa_bits) | inverted);
    return raw == 0U ? static_cast<uint16_t>(1U) : raw;
}

Here is the caller graph for this function:

score_plane_index()

__host__ __device__ constexpr uint32_t cuddl::detail::score_plane_index ( uint32_t  group, uint32_t  plane  )

constexprnoexcept

Position of plane plane of 32-bucket group group within a row's bit-planes.

A row of BucketCount 16-bit scores becomes BucketCount / 2 plane words: word p of group g holds bit p of the group's 32 scores. Lane g % 32 of a warp owns group g, and its 16 plane words form four consecutive uint4s at lane-contiguous positions, so a warp reads a row with coalesced 128-bit loads and shared copies with conflict-free 128-bit loads.

Definition at line 1334 of file kernels.cuh.

                                                                                                  {
    return (((group / 32U) * 4U + plane / 4U) * 32U + group % 32U) * 4U + plane % 4U;
}

Here is the caller graph for this function:

share_listing()

Result< std::vector< std::filesystem::path > > cuddl::detail::share_listing ( std::filesystem::path const &  input, int  rank, MPI_Comm  comm  )

inline

Rank 0's listing of input on every rank.

Definition at line 89 of file distributed.cuh.

                                                                       {
    std::string joined;
    auto listed = [&]() -> Result<void> {
        if (rank != 0) return Ok();
        auto const paths = CUDDL_TRY(list_genomes(input));
        for (auto const& path : paths) joined += path.string() + '\n';
        return Ok();
    }();
    CUDDL_TRY(agree(std::move(listed), comm));
    uint64_t length = joined.size();
    CUDDL_TRY(mpi_try(MPI_Bcast(&length, 1, MPI_UINT64_T, 0, comm), "MPI_Bcast"));
    joined.resize(length);
    // MPI counts are int: send the listing in chunks below 2 GiB.
    constexpr uint64_t chunk = uint64_t{1} << 30U;
    for (uint64_t offset = 0; offset < length; offset += chunk) {
        auto const count = static_cast<int>(std::min(chunk, length - offset));
        CUDDL_TRY(
            mpi_try(MPI_Bcast(joined.data() + offset, count, MPI_CHAR, 0, comm), "MPI_Bcast")
        );
    }
    std::vector<std::filesystem::path> paths;
    for (size_t begin = 0; begin < joined.size();) {
        auto const end = joined.find('\n', begin);
        paths.emplace_back(joined.substr(begin, end - begin));
        begin = end + 1;
    }
    return paths;
}

Here is the call graph for this function:

Here is the caller graph for this function:

shared_search()

template<typename Search >

Result< void > cuddl::detail::shared_search	(	MPI_Comm	comm,
		Search &&	search
	)

Runs search with a replica_search that shares tiles across the ranks of comm.

Definition at line 449 of file distributed.cuh.

                                                                         {
    int rank = 0;
    int size = 1;
    CUDDL_TRY(mpi_try(MPI_Comm_rank(comm, &rank), "MPI_Comm_rank"));
    CUDDL_TRY(mpi_try(MPI_Comm_size(comm, &size), "MPI_Comm_size"));
    auto counter = CUDDL_TRY(shared_tile_counter::make(comm));
    return search(
        replica_search{
            .shard = {.index = static_cast<uint32_t>(rank), .count = static_cast<uint32_t>(size)},
            .next_tile = detail::ablated("tile-queue") ? std::function<uint64_t()>{}
                                                       : [&counter] { return counter(); },
        }
    );
}

Here is the call graph for this function:

Here is the caller graph for this function:

slice_begin()

size_t cuddl::detail::slice_begin ( std::span< uint64_t const >  bytes, uint64_t  total, int  rank, int  size  )

inline

First genome of rank rank's slice: contiguous slices of about equal bytes.

Definition at line 120 of file distributed.cuh.

                                                                               {
    auto const target = static_cast<uint64_t>(static_cast<double>(total) * rank / size);
    uint64_t sum = 0;
    size_t i = 0;
    while (i < bytes.size() && sum + bytes[i] / 2 < target) sum += bytes[i++];
    return i;
}

Here is the caller graph for this function:

slice_counts()

Result< std::vector< uint32_t > > cuddl::detail::slice_counts ( reference_database_file const &  mine, genome_slice const &  slice, MPI_Comm  comm  )

inline

Every rank's genome count, after checking collectively that mine holds the sketches of this rank's slice and that every rank built with rank 0's settings.

Definition at line 235 of file distributed.cuh.

                                                                                            {
    auto const metadata = mine.metadata();
    auto reference = metadata.compatibility;
    CUDDL_TRY(mpi_try(MPI_Bcast(&reference, sizeof(reference), MPI_BYTE, 0, comm), "MPI_Bcast"));
    auto local = [&]() -> Result<void> {
        if (metadata.reference_count != slice.last - slice.first) {
            return Err(Error::invalid_argument("sketches do not match this rank's genome slice"));
        }
        if (!(reference == metadata.compatibility)) {
            return Err(Error::invalid_argument("ranks sketched with different settings"));
        }
        return Ok();
    }();
    CUDDL_TRY(agree(std::move(local), comm));
    int size = 1;
    CUDDL_TRY(mpi_try(MPI_Comm_size(comm, &size), "MPI_Comm_size"));
    uint32_t const count = metadata.reference_count;
    std::vector<uint32_t> counts(size);
    CUDDL_TRY(mpi_try(
        MPI_Allgather(&count, 1, MPI_UINT32_T, counts.data(), 1, MPI_UINT32_T, comm),
        "MPI_Allgather"
    ));
    if (std::accumulate(counts.begin(), counts.end(), uint64_t{0}) != slice.paths.size()) {
        return Err(Error::invalid_argument("ranks hold slices of different genome collections"));
    }
    return counts;
}

Here is the call graph for this function:

Here is the caller graph for this function:

smaller_index_storage()

constexpr index_storage cuddl::detail::smaller_index_storage ( uint32_t  reference_count, score_compatibility const &  compatibility  )

inlineconstexprnoexcept

Storage needing fewer device bytes for reference_count references; dense on ties.

Both store one 32-bit posting per indexed row, so only the per-storage extras are compared: dense offsets per key cell against sparse 16-bit keys and their derived key directory.

Definition at line 936 of file reference_database.cuh.

                                                                                                   {
    auto const dense = (indexed_cell_count(compatibility) + 1U) * sizeof(uint32_t);
    auto const sparse = indexed_posting_count(reference_count, compatibility) * sizeof(uint16_t) +
                        static_cast<uint64_t>(sparse_directory_entries(reference_count)) *
                            compatibility.indexed_bucket_count * sizeof(uint32_t);
    return sparse < dense ? index_storage::sparse : index_storage::dense;
}

Here is the call graph for this function:

sparse_directory_entries()

__host__ __device__ constexpr uint32_t cuddl::detail::sparse_directory_entries ( uint32_t  reference_count )

constexprnoexcept

Bounds per bucket in a sparse key directory.

Definition at line 429 of file kernels.cuh.

                                                                                                   {
    return (1U << (16U - sparse_directory_shift(reference_count))) + 1U;
}

Here is the call graph for this function:

Here is the caller graph for this function:

sparse_directory_shift()

__host__ __device__ constexpr uint32_t cuddl::detail::sparse_directory_shift ( uint32_t  reference_count )

constexprnoexcept

Low key bits a sparse key directory cell spans for reference_count references.

Cells average about eight keys per bucket for uniformly spread keys, so a lookup reads the cell's two bounds and then searches within about one memory sector.

Definition at line 420 of file kernels.cuh.

                                                                                                 {
    uint32_t cell_bits = 0U;
    while (cell_bits < 16U && (uint64_t{1} << (cell_bits + 1U)) * 8U <= reference_count) {
        ++cell_bits;
    }
    return 16U - cell_bits;
}

Here is the caller graph for this function:

splitmix64()

__host__ __device__ constexpr uint64_t cuddl::detail::splitmix64 ( uint64_t  value )

constexprnoexcept

SplitMix64 finalizer, applied to a packed k-mer XOR the domain seed.

Definition at line 9 of file hash.cuh.

                                                                           {
    value ^= value >> 30;
    value *= 0xbf58476d1ce4e5b9ULL;
    value ^= value >> 27;
    value *= 0x94d049bb133111ebULL;
    return value ^ (value >> 31);
}

Here is the caller graph for this function:

stage_groups()

template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout, typename Fill , typename Sink >

Result< void > cuddl::detail::stage_groups	(	size_t	genomes,
		uint64_t	staged_ceiling,
		std::optional< size_t >	staging_bytes,
		cuda::stream_ref	stream,
		reference_build_statistics *	statistics,
		Fill &&	fill,
		Sink &&	sink,
		bool	in_place = false,
		cuda::std::optional< blacklist_view >	blacklist = cuda::std::nullopt
	)

Runs the staging loop over a collection, filling one group at a time.

fill receives the stager, a group's first genome and its size, and stages that whole group. Genomes are numbered within their group, so a fill stages genome genome - base. sink receives each finished group's device rows with its first genome and size, and must enqueue every read of them on stream: the next group reuses the rows.

Definition at line 1111 of file database_staging.cuh.

  {
    auto const plan =
        CUDDL_TRY((plan_staging<K, BucketCount>(genomes, staged_ceiling, staging_bytes, stream)));
    database_stager<K, BucketCount, Layout> stager(stream, plan, statistics, in_place, blacklist);
    size_t base = 0;
    while (base < genomes) {
        auto const count = std::min(plan.group, genomes - base);
        CUDDL_TRY(stager.begin_group(count));
        CUDDL_TRY(fill(stager, base, count));
        auto const rows = CUDDL_TRY(stager.end_group(count));
        CUDDL_TRY(sink(rows, base, count));
        base += count;
    }
    return stager.finish();
}

Here is the call graph for this function:

stage_paths()

template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout, typename Sink >

Result< void > cuddl::detail::stage_paths	(	std::span< std::filesystem::path const >	paths,
		cuda::stream_ref	stream,
		std::optional< size_t >	staging_bytes,
		unsigned	parser_workers,
		transfer_mode	transfer,
		decompression_backend	decompression,
		bool	direct_reads,
		reference_build_statistics *	statistics,
		Sink &&	sink,
		cuda::std::optional< blacklist_view >	blacklist = cuda::std::nullopt
	)

Stages a path collection, handing each group's device rows to sink.

parser_workers is a value, not a sentinel: the build clamps it to what the inputs and the hardware allow, so asking for the default is asking for the default.

The arena ceiling is the decompressed size of every input: a bound from compressed sizes alone overestimates a corpus several times over, and an arena past what the corpus can hold is memory the device never needs.

Definition at line 1612 of file database_staging.cuh.

  {
    // Loaders and inflation reuse pinned buffers through the build; none outlive it.
    pinned_pool_trim const release_pinned{stream.device(), pinned_placement::host};
    if (parser_workers == 0) {
        return Err(Error::invalid_argument("a path build needs at least one loader"));
    }
#if !CUDDL_HAS_NVCOMP
    if (decompression == decompression_backend::gpu ||
        decompression == decompression_backend::gpu_cores ||
        decompression == decompression_backend::hybrid) {
        return Err(
            Error::invalid_argument("requested decompression backend requires nvCOMP support")
        );
    }
    decompression = decompression_backend::cpu;
#endif
    if (transfer == transfer_mode::in_place && !device_reads_pageable_memory(stream.device())) {
        return Err(
            Error::invalid_argument(
                "in-place staging needs a device that reads pageable host memory"
            )
        );
    }
    if (paths.empty()) return Ok();
    // A coherent GPU reads the loaders' buffers in place, so CPU inflation feeds it best.
    // Elsewhere nvCOMP wins: 100,000 genomes on an RTX 5090 with 120 loaders built in 7.4 s with
    // gpu and 10.2 s with cpu, and the CPU-inflating backends kept the GPU 60% idle.
    if (decompression == decompression_backend::automatic) {
        if (device_reads_pageable_memory(stream.device())) {
            decompression = path_loaders::worker_count(paths.size(), parser_workers) > 1
                                ? decompression_backend::hybrid
                                : decompression_backend::cpu;
        } else {
            decompression = decompression_backend::gpu;
        }
    }
#if CUDDL_HAS_NVCOMP
    if (decompression == decompression_backend::hybrid) {
        return stage_paths_normalising<K, BucketCount, Layout>(
            paths,
            stream,
            staging_bytes,
            parser_workers,
            transfer,
            direct_reads,
            statistics,
            std::forward<Sink>(sink),
            blacklist
        );
    }
    if (decompression != decompression_backend::cpu) {
        return stage_paths_inflating<K, BucketCount, Layout>(
            paths,
            stream,
            staging_bytes,
            parser_workers,
            transfer,
            direct_reads,
            decompression == decompression_backend::gpu,
            statistics,
            std::forward<Sink>(sink),
            blacklist
        );
    }
#endif
    auto const in_place = stages_in_place(transfer, stream.device());
    auto const page_locked = pages_locked(transfer, stream.device());
    // Copied bases cross the bus, and packing them first moves a quarter of the bytes. Unpacked,
    // an A100 took 2.0-2.2 s for 10,000 genomes from 32 to 112 loaders, held by the copies;
    // packed, 1.8 s with 32 and 1.3 s with 64 or more. Packing costs the loaders time, so with
    // 16 it took 3.4 s against 2.8 s. A device that reads the bases in place gains nothing from
    // it: 16 loaders on a GH200 took 3.0 s packed against 2.7 s without.
    path_loaders loaders(
        stream,
        paths,
        parser_workers,
        page_locked,
        in_place,
        in_place || ablated("base-packing") ? 0 : K,
        direct_reads
    );
    // The corpus size only bounds a default arena, and only below its ceiling, so probing stops
    // there: a corpus of any size then costs at most a few dozen file opens here.
    uint64_t staged_ceiling = 0;
    for (auto const& path : paths) {
        if (staging_bytes.has_value() || staged_ceiling >= default_arena_ceiling) break;
        auto const decompressed = gzip_decompressed_size(path.string());
        if (decompressed != 0) {
            staged_ceiling += decompressed;
            continue;
        }
        std::error_code error;
        auto const size = std::filesystem::file_size(path, error);
        if (!error) staged_ceiling += size;
    }
    CUDDL_TRY((stage_groups<K, BucketCount, Layout>(
        paths.size(),
        staged_ceiling,
        staging_bytes,
        stream,
        statistics,
        [&](database_stager<K, BucketCount, Layout>& stager, size_t base, size_t count)
            -> Result<void> {
            for (size_t id = base; id < base + count; ++id) {
                auto sequence = CUDDL_TRY(loaders.take(paths, id));
                // Read the loaded file's parts before the move below: argument order is
                // unspecified, so a moved-from Result must not be dereferenced.
                auto const* const pinned_base =
                    page_locked ? sequence->decompressed_target : nullptr;
                auto const pinned_size = page_locked ? sequence->decompressed_size : 0;
                auto const& extents = sequence->extents;
                if (sequence->words != nullptr) {
                    CUDDL_TRY(stager.add_packed_genome(
                        id - base, std::move(sequence), pinned_base, pinned_size
                    ));
                    continue;
                }
                CUDDL_TRY(stager.add_genome(
                    id - base, extents, pinned_base, pinned_size, std::move(sequence)
                ));
            }
            return Ok();
        },
        sink,
        in_place,
        blacklist
    )));
    if (statistics != nullptr) {
        statistics->pinned_buffers = loaders.page_locked_buffers();
        statistics->workers = static_cast<unsigned>(loaders.workers());
        statistics->in_place = in_place;
    }
    loaders.reset();
    return Ok();
}

Here is the call graph for this function:

stage_sequences()

template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout, typename Sink >

Result< void > cuddl::detail::stage_sequences	(	std::span< sequence_genome const >	genomes,
		cuda::stream_ref	stream,
		std::optional< size_t >	staging_bytes,
		reference_build_statistics *	statistics,
		Sink &&	sink,
		cuda::std::optional< blacklist_view >	blacklist = cuda::std::nullopt
	)

Stages bases the caller already holds, handing each group's device rows to sink.

Definition at line 1761 of file database_staging.cuh.

  {
    if (genomes.empty()) return Ok();
    pinned_pool_trim const release_pinned{stream.device(), pinned_placement::host};
    // The caller knows exactly what the corpus holds, so the ceiling is the sum of its bases
    // rather than an estimate read out of file headers.
    uint64_t staged_ceiling = 0;
    for (auto const& genome : genomes) {
        for (auto const& record : genome.records) staged_ceiling += record.bases.size();
    }
    std::vector<fastx_sequence_extent> records;
    return stage_groups<K, BucketCount, Layout>(
        genomes.size(),
        staged_ceiling,
        staging_bytes,
        stream,
        statistics,
        [&](database_stager<K, BucketCount, Layout>& stager,
            size_t base,
            size_t count) -> Result<void> {
            for (size_t id = base; id < base + count; ++id) {
                auto const& genome = genomes[id];
                records.clear();
                records.reserve(genome.records.size());
                for (auto const& record : genome.records) {
                    if (record.bases.empty()) continue;
                    records.push_back(
                        {record.bases.data(), record.bases.data() + record.bases.size()}
                    );
                }
                CUDDL_TRY(stager.add_genome(id - base, records));
            }
            return Ok();
        },
        sink,
        false,
        blacklist
    );
}

Here is the call graph for this function:

Here is the caller graph for this function:

stages_in_place()

bool cuddl::detail::stages_in_place ( transfer_mode  mode, cuda::device_ref  device  )

inlinenoexcept

Whether a build in mode stages the caller's bytes in place.

Automatic transfer reads CPU-decoded buffers in place on coherent devices.

Definition at line 90 of file database_staging.cuh.

                                                                                              {
    return mode == transfer_mode::in_place ||
           (mode == transfer_mode::automatic && device_reads_pageable_memory(device));
}

Here is the call graph for this function:

Here is the caller graph for this function:

sum_posting_bitmaps()

template<uint32_t CounterBits>

__device__ uint32_t cuddl::detail::sum_posting_bitmaps	(	uint32_t const *	bitmaps,
		uint32_t const *	starts,
		uint32_t	count,
		uint32_t	word,
		uint32_t	minimum_matches,
		uint32_t	passed,
		uint32_t(&)	counters[CounterBits]
	)

Sums one query's posting bitmaps over one word of references in bit-sliced counters.

Independent bitmap loads overlap in batches. Returns passed with every reference whose count reaches minimum_matches added; summing stops once all 32 have passed, so a passed reference's count is unused and may wrap.

Definition at line 844 of file kernels.cuh.

  {
    constexpr uint32_t batch = 8U;
    for (uint32_t i = 0U; i < count && passed != ~0U; i += batch) {
        // Keep independent bitmap reads in flight before consuming their values.
        uint32_t loaded[batch]{};
        _Pragma("unroll")
        for (uint32_t j = 0U; j < batch; ++j) {
            if (i + j < count) {
                loaded[j] = bitmaps[starts[i + j] + word];
            }
        }
        // Reduce eight bitmaps to four input count planes.
        auto const sum0 = loaded[0] ^ loaded[1] ^ loaded[2];
        auto const carry0 = (loaded[0] & loaded[1]) | ((loaded[0] | loaded[1]) & loaded[2]);
        auto const sum1 = loaded[3] ^ loaded[4] ^ loaded[5];
        auto const carry1 = (loaded[3] & loaded[4]) | ((loaded[3] | loaded[4]) & loaded[5]);
        auto const ones0 = sum0 ^ sum1 ^ loaded[6];
        auto const carry2 = (sum0 & sum1) | ((sum0 | sum1) & loaded[6]);
        auto const twos0 = carry0 ^ carry1 ^ carry2;
        auto const fours0 = (carry0 & carry1) | ((carry0 | carry1) & carry2);
        auto const carry3 = ones0 & loaded[7];
        auto const carry4 = twos0 & carry3;
        uint32_t const additions[]{
            ones0 ^ loaded[7], twos0 ^ carry3, fours0 ^ carry4, fours0 & carry4
        };
        uint32_t carry = 0U;
        uint32_t at_least = ~0U;
        _Pragma("unroll")
        for (uint32_t bit = 0U; bit < CounterBits; ++bit) {
            auto const extra = bit < 4U ? additions[bit] : 0U;
            auto const previous = counters[bit];
            counters[bit] = previous ^ carry ^ extra;
            carry = (previous & carry) | ((previous | carry) & extra);
            auto const key = 0U - ((minimum_matches >> bit) & 1U);
            at_least = (counters[bit] & ~key) | ((counters[bit] | ~key) & at_least);
        }
        // Overflow is above every threshold representable by these counter planes.
        passed |= at_least | carry;
        _Pragma("unroll")
        for (uint32_t bit = CounterBits; bit < 4U; ++bit) {
            passed |= additions[bit];
        }
    }
    return passed;
}

Here is the call graph for this function:

Here is the caller graph for this function:

top_k_heap_words()

constexpr size_t cuddl::detail::top_k_heap_words ( uint32_t  k )

constexprnoexcept

Words of shared memory one heap takes: k keys, then their k packed counts.

Definition at line 56 of file top_k.cuh.

                                                                     {
    return size_t{k} * 2U;
}

Here is the caller graph for this function:

top_k_key()

__host__ __device__ constexpr uint64_t cuddl::detail::top_k_key ( pairwise_counts const &  counts, uint32_t  id  )

constexprnoexcept

Ranking key of a neighbour: its weighted k-mer identity above its complemented ID.

Keys are unique, a larger key ranks higher, and equal identities rank the smaller ID higher, so any selection of the k largest keys is the same set. Zero means no candidate; a real key is never zero because IDs stay below UINT32_MAX.

Definition at line 29 of file top_k.cuh.

                                                               {
    auto const identity = cuda::std::bit_cast<uint32_t>(weighted_kmer_identity(counts));
    return (uint64_t{identity} << 32U) | static_cast<uint32_t>(~id);
}

Here is the call graph for this function:

Here is the caller graph for this function:

top_k_nth_bit()

__device__ uint32_t cuddl::detail::top_k_nth_bit ( uint32_t  mask, uint32_t  n  )

inline

Position of set bit n, counting from zero, of mask, which has more than n set bits.

Definition at line 114 of file top_k.cuh.

                                                                    {
    uint32_t position = 0U;
    asm("fns.b32 %0, %1, 0, %2;" : "=r"(position) : "r"(mask), "r"(static_cast<int>(n + 1U)));
    return position;
}

Here is the caller graph for this function:

top_k_replace_root()

__device__ void cuddl::detail::top_k_replace_root ( uint64_t *  heap, uint32_t  k, uint64_t  key, uint64_t  value  )

inline

Replaces the root of a min-heap of k keys at heap, whose values follow the keys, with key and value, and sifts it down.

The caller owns the heap and has checked that key beats the root.

Definition at line 66 of file top_k.cuh.

                                                                             {
    auto* const values = heap + k;
    uint32_t at = 0U;
    for (auto child = 1U; child < k; child = 2U * at + 1U) {
        auto child_key = heap[child];
        if (child + 1U < k && heap[child + 1U] < child_key) child_key = heap[++child];
        if (child_key >= key) break;
        heap[at] = child_key;
        values[at] = values[child];
        at = child;
    }
    heap[at] = key;
    values[at] = value;
}

Here is the caller graph for this function:

top_k_stored()

template<bool UpperTriangle>

__device__ packed_pairwise_counts cuddl::detail::top_k_stored ( batch_result_tile const &  tile, uint32_t  query, uint32_t  reference  )

inline

The stored result of tile row query and reference, which must pass.

Definition at line 106 of file top_k.cuh.

                                                                                {
    auto const slot = batch_result_slot<UpperTriangle>(
        query, reference, tile.first_query_id, tile.reference_count
    );
    return packed_pairwise_counts{__ldg(&tile.results[slot].bits)};
}

top_k_warp_copy()

template<bool ToShared>

__device__ void cuddl::detail::top_k_warp_copy ( top_k_slots const &  slots, size_t  row, uint64_t *  heap  )

inline

Copies the k entries of global heap row into, or for ToShared false out of, the shared heap heap with the warp's lanes.

Definition at line 177 of file top_k.cuh.

                                                                                             {
    auto const k = slots.k;
    for (auto i = static_cast<uint32_t>(threadIdx.x) % 32U; i < k; i += 32U) {
        if constexpr (ToShared) {
            heap[i] = slots.keys[row * k + i];
            heap[k + i] = slots.counts[row * k + i];
        } else {
            slots.keys[row * k + i] = heap[i];
            slots.counts[row * k + i] = heap[k + i];
        }
    }
    __syncwarp();
}

top_k_warp_copy< false >()

cuddl::detail::top_k_warp_copy< false >	(	slots	,
		row	,
		heap
	)

top_k_warp_copy< true >()

cuddl::detail::top_k_warp_copy< true >	(	slots	,
		row	,
		heap
	)

top_k_warp_merge() [1/2]

cuddl::detail::top_k_warp_merge	(	heap	,
		slots.	k,
		words -	cuda::std::minskipped, words,
		[&] (uint32_t word) { return top_k_word_bits< Thresholded, UpperTriangle >(tile, query, skipped+word, words);}	,
		[&] (uint32_t candidate, uint64_t &key, uint64_t &value) { auto const reference=skipped *32U+candidate;auto const stored=top_k_stored< UpperTriangle >(tile, query, reference);key=top_k_key(stored.unpack(tile.bucket_count), reference);value=stored.bits;}
	)

top_k_warp_merge() [2/2]

template<typename Bits , typename Entry >

__device__ void cuddl::detail::top_k_warp_merge ( uint64_t *  heap, uint32_t  k, uint32_t  words, Bits  bits_of, Entry  entry_of  )

inline

One warp merges the candidates of words bitmap words into the heap of k entries at heap, in shared memory, which only this warp updates, so it needs no lock.

bits_of gives word w's candidates and entry_of candidate w * 32 + bit's key and value. The warp takes 32 words at a time and hands their set bits out 32 per step, one per lane in order, so every step offers 32 candidates however the words are filled. Lanes test their candidates against the root; the warp then inserts the survivors one at a time.

Definition at line 131 of file top_k.cuh.

                                                                                           {
    auto const lane = static_cast<uint32_t>(threadIdx.x) % 32U;
    for (uint32_t base = 0U; base < words; base += 32U) {
        auto const bits = base + lane < words ? bits_of(base + lane) : 0U;
        // Inclusive prefix of the lanes' set bits, so candidate j is in the first lane whose
        // prefix exceeds j.
        auto const count = static_cast<uint32_t>(cuda::std::popcount(bits));
        auto prefix = count;
        _Pragma("unroll")
        for (uint32_t offset = 1U; offset < 32U; offset *= 2U) {
            auto const below = __shfl_up_sync(~0U, prefix, offset);
            if (lane >= offset) prefix += below;
        }
        auto const total = __shfl_sync(~0U, prefix, 31);
        for (uint32_t first = 0U; first < total; first += 32U) {
            auto const candidate = first + lane;
            // Every lane takes part in the search's shuffles, with or without a candidate.
            uint32_t source = 0U;
            _Pragma("unroll")
            for (uint32_t step = 16U; step != 0U; step /= 2U) {
                if (__shfl_sync(~0U, prefix, source + step - 1U) <= candidate) source += step;
            }
            auto const source_bits = __shfl_sync(~0U, bits, source);
            auto const before = __shfl_sync(~0U, prefix - count, source);
            uint64_t key = 0U, value = 0U;
            if (candidate < total) {
                entry_of(
                    (base + source) * 32U + top_k_nth_bit(source_bits, candidate - before),
                    key,
                    value
                );
            }
            for (auto pending = __ballot_sync(~0U, key > heap[0]); pending != 0U;
                 pending &= pending - 1U) {
                if (lane == static_cast<uint32_t>(__ffs(pending) - 1) && key > heap[0]) {
                    top_k_replace_root(heap, k, key, value);
                }
                __syncwarp();
            }
        }
    }
}

Here is the call graph for this function:

top_k_word_bits()

template<bool Thresholded, bool UpperTriangle>

__device__ uint32_t cuddl::detail::top_k_word_bits ( batch_result_tile const &  tile, uint32_t  query, uint32_t  word, uint32_t  words  )

inline

The pass bits of word word of tile row query: the stored bits, or for exhaustive tiles every reference of the word.

Pairs on or below the diagonal of all-to-all tiles are masked.

Definition at line 85 of file top_k.cuh.

                                                                                              {
    uint32_t bits = 0U;
    if constexpr (Thresholded) {
        bits = __ldg(tile.pass_bits + static_cast<size_t>(query) * words + word);
    } else {
        auto const left = tile.reference_count - word * 32U;
        bits = left >= 32U ? ~0U : (1U << left) - 1U;
    }
    if constexpr (UpperTriangle) {
        auto const query_id = tile.first_query_id + query;
        if (word * 32U <= query_id) {
            auto const skipped = query_id - word * 32U + 1U;
            bits &= skipped >= 32U ? 0U : ~0U << skipped;
        }
    }
    return bits;
}

try_unwrap_success() [1/2]

template<typename T >

T cuddl::detail::try_unwrap_success

(

Result< T > &

result

)

Definition at line 160 of file error.hpp.

                                                      {
    return std::move(*result);
}

try_unwrap_success() [2/2]

void cuddl::detail::try_unwrap_success ( Result< void > &  result )

inline

Definition at line 164 of file error.hpp.

                                                     {
    (void)result;
}

uses_tiled_index_counts()

bool cuddl::detail::uses_tiled_index_counts ( uint32_t  reference_count, uint32_t  query_count, uint32_t  indexed_bucket_count  )

inlinenoexcept

Whether a batch counts through count_batch_index_tile_kernel.

Its 16-bit counters hold at most 65535 matches. Tiny batches finish sooner on the global kernel, which needs no per-block counter initialization or flush.

Definition at line 691 of file kernels.cuh.

           {
    return reference_count != 0U && indexed_bucket_count <= 0xffffU &&
           (reference_count >= 8192U || query_count >= 512U);
}

validate_indexed_score_compatibility()

template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout>

Result< void > cuddl::detail::validate_indexed_score_compatibility ( score_compatibility const &  compatibility )

inline

Definition at line 906 of file reference_database.cuh.

  {
    CUDDL_TRY((validate_score_compatibility<K, BucketCount, Layout>(compatibility)));
    auto const full_bucket_count = static_cast<uint32_t>(BucketCount);
    if (compatibility.indexed_bucket_count != full_bucket_count &&
        compatibility.indexed_bucket_count != full_bucket_count / 2U) {
        return Err(Error::invalid_argument("indexed builds require a full or half bucket count"));
    }
    if (compatibility.key_mask != 0xffffU && compatibility.key_mask != 0x7fffU) {
        return Err(Error::invalid_argument("indexed builds require a 16-bit or 15-bit key mask"));
    }
    return Ok();
}

Here is the call graph for this function:

validate_non_indexed_score_compatibility()

template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout>

Result< void > cuddl::detail::validate_non_indexed_score_compatibility ( score_compatibility const &  compatibility )

inline

Definition at line 892 of file reference_database.cuh.

  {
    CUDDL_TRY((validate_score_compatibility<K, BucketCount, Layout>(compatibility)));
    if (compatibility.indexed_bucket_count != BucketCount) {
        return Err(Error::invalid_argument("non-indexed builds require every bucket"));
    }
    if (compatibility.key_mask != 0xffffU && compatibility.key_mask != 0x7fffU) {
        return Err(Error::invalid_argument("non-indexed builds require a 16-bit or 15-bit key"));
    }
    return Ok();
}

Here is the call graph for this function:

validate_score_compatibility()

template<uint32_t K, size_t BucketCount, typename Layout = default_register_layout>

Result< void > cuddl::detail::validate_score_compatibility ( score_compatibility const &  compatibility )

inline

Definition at line 868 of file reference_database.cuh.

  {
    if (compatibility.kmer_length != K) {
        return Err(Error::invalid_argument("k-mer length does not match the database type"));
    }
    if (compatibility.bucket_count != BucketCount) {
        return Err(Error::invalid_argument("bucket count does not match the database type"));
    }
    if (compatibility.score_encoder_identity == 0U ||
        compatibility.exponent_bits != Layout::exponent_bits ||
        compatibility.mantissa_bits != Layout::mantissa_bits) {
        return Err(Error::invalid_argument("score encoding does not match the register layout"));
    }
    if (compatibility.hash_identity == 0U) {
        return Err(Error::invalid_argument("hash identity must be specified"));
    }
    if (compatibility.canonicalisation_policy == 0U) {
        return Err(Error::invalid_argument("canonicalisation policy must be specified"));
    }
    return Ok();
}

Here is the call graph for this function:

walk_bgzf()

template<typename Block >

std::optional< std::pair< uint32_t, uint64_t > > cuddl::detail::walk_bgzf	(	unsigned char const *	bytes,
		uint64_t	size,
		Block &&	block
	)

Calls block(deflate_at, deflate_bytes, inflated) for each non-empty block of the BGZF file in bytes.

Returns

The non-empty blocks and their inflated total, or nullopt unless whole blocks tile all size bytes.

Definition at line 111 of file device_gzip.cuh.

                                                                    {
    uint64_t at = 0, total = 0;
    uint32_t count = 0;
    while (at < size) {
        auto const block_size = size - at >= 18 ? bgzf_block_size(bytes + at) : 0;
        if (block_size < 26 || block_size > size - at) return std::nullopt;
        uint32_t inflated = 0;
        std::memcpy(&inflated, bytes + at + block_size - 4, 4);
        if (inflated != 0) {
            block(at + 18, uint64_t{block_size} - 26, inflated);
            total += inflated;
            ++count;
        }
        at += block_size;
    }
    return std::pair{count, total};
}

Here is the call graph for this function:

Here is the caller graph for this function:

warp_grid_blocks()

uint32_t cuddl::detail::warp_grid_blocks ( size_t  warps )

inlinenoexcept

Blocks for a grid-stride launch covering warps warps of block_size threads.

Definition at line 1524 of file kernels.cuh.

                                                                      {
    constexpr size_t warps_per_block = block_size / 32U;
    return static_cast<uint32_t>(
        cuda::std::min<size_t>((warps + warps_per_block - 1U) / warps_per_block, 65535U)
    );
}

Here is the caller graph for this function:

wide_rows_aligned()

template<typename QueryScore , typename ReferenceScore >

__host__ __device__ bool cuddl::detail::wide_rows_aligned ( QueryScore const *  query, ReferenceScore const *  reference  )

inlinenoexcept

True when both row bases meet load_256_alignment on the compiled architecture.

Definition at line 33 of file helpers.cuh.

                                                                                     {
    constexpr uintptr_t mask = load_256_alignment() - 1U;
    return (reinterpret_cast<uintptr_t>(query) & mask) == 0U &&
           (reinterpret_cast<uintptr_t>(reference) & mask) == 0U;
}

Here is the call graph for this function:

write_score_planes()

__device__ __forceinline__ void cuddl::detail::write_score_planes	(	uint32_t	score,
		uint32_t	group,
		uint32_t	lane,
		uint32_t *	planes
	)

Writes the bit-planes of one 32-bucket group whose scores the warp holds, one per lane.

Definition at line 1340 of file kernels.cuh.

                                                                                    {
    uint32_t mine = 0U;
    _Pragma("unroll")
    for (uint32_t plane = 0; plane < score_planes; ++plane) {
        auto const word = __ballot_sync(0xffffffffU, ((score >> plane) & 1U) != 0U);
        if (lane == plane) {
            mine = word;
        }
    }
    if (lane < score_planes) {
        planes[score_plane_index(group, lane)] = mine;
    }
}

Here is the call graph for this function:

xxhash64() [1/2]

template<typename T >

__host__ __device__ uint64_t cuddl::detail::xxhash64 ( const T &  key, uint64_t  seed = 0  )

inline

Computes xxHash-64 over the raw bytes of a value.

Definition at line 155 of file hash.cuh.

                                                                              {
    return xxhash64(reinterpret_cast<const uint8_t*>(&key), sizeof(T), seed);
}

Here is the call graph for this function:

xxhash64() [2/2]

__host__ __device__ uint64_t cuddl::detail::xxhash64 ( const uint8_t *  bytes, uint64_t  size, uint64_t  seed  )

inline

Computes xxHash-64 over a byte range without allocating or copying it.

Definition at line 63 of file hash.cuh.

                                                                                                 {
    uint64_t offset = 0;
    uint64_t h64;
 
    // Process 32-byte chunks
    if (size >= 32) {
        uint64_t limit = size - 32;
        uint64_t v1 = seed + PRIME64_1 + PRIME64_2;
        uint64_t v2 = seed + PRIME64_2;
        uint64_t v3 = seed;
        uint64_t v4 = seed - PRIME64_1;
 
        do {
            const uint64_t pipeline_offset = offset / 8;
            v1 += load_chunk<uint64_t>(bytes, pipeline_offset + 0) * PRIME64_2;
            v1 = rotl64(v1, 31);
            v1 *= PRIME64_1;
            v2 += load_chunk<uint64_t>(bytes, pipeline_offset + 1) * PRIME64_2;
            v2 = rotl64(v2, 31);
            v2 *= PRIME64_1;
            v3 += load_chunk<uint64_t>(bytes, pipeline_offset + 2) * PRIME64_2;
            v3 = rotl64(v3, 31);
            v3 *= PRIME64_1;
            v4 += load_chunk<uint64_t>(bytes, pipeline_offset + 3) * PRIME64_2;
            v4 = rotl64(v4, 31);
            v4 *= PRIME64_1;
            offset += 32;
        } while (offset <= limit);
 
        h64 = rotl64(v1, 1) + rotl64(v2, 7) + rotl64(v3, 12) + rotl64(v4, 18);
 
        v1 *= PRIME64_2;
        v1 = rotl64(v1, 31);
        v1 *= PRIME64_1;
        h64 ^= v1;
        h64 = h64 * PRIME64_1 + PRIME64_4;
 
        v2 *= PRIME64_2;
        v2 = rotl64(v2, 31);
        v2 *= PRIME64_1;
        h64 ^= v2;
        h64 = h64 * PRIME64_1 + PRIME64_4;
 
        v3 *= PRIME64_2;
        v3 = rotl64(v3, 31);
        v3 *= PRIME64_1;
        h64 ^= v3;
        h64 = h64 * PRIME64_1 + PRIME64_4;
 
        v4 *= PRIME64_2;
        v4 = rotl64(v4, 31);
        v4 *= PRIME64_1;
        h64 ^= v4;
        h64 = h64 * PRIME64_1 + PRIME64_4;
    } else {
        h64 = seed + PRIME64_5;
    }
 
    h64 += size;
 
    // Process remaining 8-byte chunks
    if ((size % 32) >= 8) {
        for (; offset <= size - 8; offset += 8) {
            uint64_t k1 = load_chunk<uint64_t>(bytes, offset / 8) * PRIME64_2;
            k1 = rotl64(k1, 31) * PRIME64_1;
            h64 ^= k1;
            h64 = rotl64(h64, 27) * PRIME64_1 + PRIME64_4;
        }
    }
 
    // Process remaining 4-byte chunks
    if ((size % 8) >= 4) {
        for (; offset <= size - 4; offset += 4) {
            h64 ^= (load_chunk<uint32_t>(bytes, offset / 4) & 0xffffffffULL) * PRIME64_1;
            h64 = rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
        }
    }
 
    // Process remaining bytes
    if (size % 4) {
        while (offset < size) {
            h64 ^= (bytes[offset] & 0xff) * PRIME64_5;
            h64 = rotl64(h64, 11) * PRIME64_1;
            ++offset;
        }
    }
 
    return finalize(h64);
}

Here is the call graph for this function:

Here is the caller graph for this function:

Variable Documentation

ablations

constexpr std::array<std::string_view, 22> cuddl::detail::ablations

inlineconstexpr

Initial value:

{
    "co-scheduling",       
    "read-depth",          
    "probe-depth",         
    "pinned-pool",         
    "numa-pool",           
    "pinned-reservation",  
    "input-lanes",         
    "simd-parsing",        
    "base-packing",        
    "kmer-encoding",       
    "deltas",              
    "bit-slicing",         
    "lop3-kernel",         
    "tile-queue",          
    "progress-thread",     
    "match-filter",        
    "posting-bitmaps",     
    "index-fallback",      
    "inverted-index",      
    "direct-io",           
    "io-uring",            
    "write-probe",         
}

Optimisations that the CUDDL_ABLATE environment variable can turn off in a build configured with -Dablations=true (CUDDL_ABLATIONS=1).

They exist to measure what each optimisation contributes (scripts/ablation.py); results are the same either way. Other builds compile every switch out.

Definition at line 19 of file ablation.hpp.

                                                       {
    "co-scheduling",       // nvCOMP claims a batch only if it finishes before the host loaders
    "read-depth",          // readers keep parallel_read_depth reads in flight, not one per loader
    "probe-depth",         // 4 * parallel_read_depth gzip probes in flight, not parallel_read_depth
    "pinned-pool",         // device-only pinned pools that keep their pages, not CCCL-like ones
    "numa-pool",           // device-written pinned buffers on the device's host NUMA node
    "pinned-reservation",  // packed-word buffers pinned before the loaders start
    "input-lanes",         // GPU lanes sized from the input, not a fixed 1 GiB
    "simd-parsing",        // AVX2/AVX-512/NEON line scans, compaction, packing and gzip scans
    "base-packing",        // the CPU path packs bases into two-bit words before copying them
    "kmer-encoding",       // the GPU builds eight windows per thread from shared SWAR-packed tiles
    "deltas",              // near-duplicate rows corrected from a base row, not compared in full
    "bit-slicing",         // exhaustive search on bit-planes, not packed row-major 16-bit scores
    "lop3-kernel",         // the tiled LOP3 plane kernel, not one bucket per lane step
    "tile-queue",          // ranks claim all-to-all tiles from a shared counter, not a fixed split
    "progress-thread",     // rank 0 drives MPI progress for the shared tile counter
    "match-filter",        // a low-plane bound selects candidates before exact refinement
    "posting-bitmaps",     // long posting lists of the inverted index are kept as bitmaps
    "index-fallback",      // an index whose lists cover too many pairs is bypassed
    "inverted-index",      // threshold searches use an inverted index (benchmarks only)
    "direct-io",           // result files are written with O_DIRECT, not through the page cache
    "io-uring",            // direct result writes keep several blocks in flight with io_uring
    "write-probe",         // writers that need not be durable use the page cache while it is faster
};

ablations_enabled

constexpr bool cuddl::detail::ablations_enabled = false

inlineconstexpr

Definition at line 79 of file ablation.hpp.

all_pairs_block_size

constexpr uint32_t cuddl::detail::all_pairs_block_size = 512U

constexpr

Threads per refine_all_pairs_kernel block.

Two references' planes stay in registers, so a block holds fewer threads than a refine block.

Definition at line 2299 of file kernels.cuh.

all_pairs_word_parts

constexpr uint32_t cuddl::detail::all_pairs_word_parts = 4U

constexpr

Parts a refine_all_pairs_kernel warp claims each 32-reference word in.

Warps idle at the end of a cell until the slowest one finishes its last claim, and words differ in cost, so smaller claims shorten that wait. On a GPU running at its power limit the shorter wait only lowers the clock; with power headroom it shortens the search.

Definition at line 2305 of file kernels.cuh.

all_to_all

static __global__ uint32_t uint32_t uint32_t uint32_t bool cuddl::detail::all_to_all

Definition at line 928 of file kernels.cuh.

batch_query_tile_count

constexpr uint32_t cuddl::detail::batch_query_tile_count = 128U

constexpr

Minimum query rows per batch tile.

Definition at line 955 of file reference_database.cuh.

bitmap_counts

__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool uint32_t uint32_t const uint32_t const uint32_t const* cuddl::detail::bitmap_counts

Initial value:

{
    constexpr uint32_t warp_width = 32

Definition at line 932 of file kernels.cuh.

bitmap_refine_block_size

constexpr uint32_t cuddl::detail::bitmap_refine_block_size = 768U

constexpr

Threads per refine_batch_bitmap_kernel block.

Leaves register space for query-plane loads alongside cached reference planes.

Definition at line 1649 of file kernels.cuh.

bitmap_refine_dynamic_bytes

template<size_t BucketCount>

constexpr size_t cuddl::detail::bitmap_refine_dynamic_bytes

constexpr

Initial value:

=
    bitmap_refine_staged<BucketCount>
        ? static_cast<size_t>(bitmap_refine_group<BucketCount>) * BucketCount * sizeof(uint16_t)
        : 0U

Shared bytes one refine_batch_bitmap_kernel block stages.

Definition at line 1680 of file kernels.cuh.

bitmap_refine_group

template<size_t BucketCount>

constexpr uint32_t cuddl::detail::bitmap_refine_group

constexpr

Initial value:

=
    bitmap_refine_staged<BucketCount>
        ? static_cast<uint32_t>(cuda::std::min<size_t>(
              24U,
              bitmap_refine_shared_bytes / (BucketCount * sizeof(uint16_t))
          ))
        : 8U

Queries one refine_batch_bitmap_kernel block refines together.

Each reference row is read once per group, so the group size divides the kernel's L2 traffic. Rows too large to stage in shared memory are read from global memory in groups of 8.

Definition at line 1670 of file kernels.cuh.

bitmap_refine_shared_bytes

constexpr size_t cuddl::detail::bitmap_refine_shared_bytes = 96U * 1024U

constexpr

Shared bytes refine_batch_bitmap_kernel may stage query planes in; one block per SM fits on every supported architecture.

Definition at line 1659 of file kernels.cuh.

bitmap_refine_staged

template<size_t BucketCount>

constexpr bool cuddl::detail::bitmap_refine_staged = bitmap_refine_shared_bytes >= BucketCount * sizeof(uint16_t)

constexpr

Whether a query group's planes fit bitmap_refine_shared_bytes.

Definition at line 1663 of file kernels.cuh.

bitmap_refine_waves

constexpr uint32_t cuddl::detail::bitmap_refine_waves = 8U

constexpr

Cells per refine_batch_bitmap_kernel block that a tile of bitmap-selected pairs is split into when it allows.

Selected pairs cluster around near-duplicate genomes, so cells differ in work by 20 times or more; with one cell per block the densest cell sets the kernel's time.

Definition at line 1655 of file kernels.cuh.

bitmap_starts

__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool uint32_t uint32_t const uint32_t const* cuddl::detail::bitmap_starts

Definition at line 931 of file kernels.cuh.

bitmaps

__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t uint32_t uint16_t const uint2 const uint32_t uint32_t bool uint32_t uint32_t const* cuddl::detail::bitmaps

Definition at line 930 of file kernels.cuh.

bits

auto const cuddl::detail::bits

Initial value:

= candidate_bits +
                       static_cast<size_t>(query_index) * candidate_bit_words(reference_count) +
                       low / 32U

Definition at line 1184 of file kernels.cuh.

blacklist

__global__ size_t uint32_t bool blacklist_view cuddl::detail::blacklist

Definition at line 96 of file kernels.cuh.

                               {}
) {
    __shared__ uint32_t state[BucketCount];
    for (auto i = threadIdx.x; i < BucketCount; i += blockDim.x) {
        state[i] = 0U;
    }
    __syncthreads();
 
    auto const process = [&](uint64_t value) {
        auto const hash = hash_kmer(value);
        auto const incoming = static_cast<uint32_t>(score<Layout>(hash));
        auto const bucket = bucket_of<BucketCount>(hash);
        if constexpr (HasBlacklist) {
            if (blacklist.contains(value, hash, bucket)) return;
        }
        atomicMax(&state[bucket], incoming);
    };
 
    auto const stride = static_cast<size_t>(gridDim.x) * blockDim.x * 4U;
    auto const index = (static_cast<size_t>(blockIdx.x) * blockDim.x + threadIdx.x) * 4U;
    for (auto offset = index; offset < input_size; offset += stride) {
        if (vector_input && offset + 4U <= input_size) {
            uint64_t values[4];
            load_256_global_nc(input + offset, values);
            _Pragma("unroll")
            for (uint32_t item = 0; item < 4U; ++item) {
                process(values[item]);
            }
        } else {
            // Scalar path: unaligned input keeps striding exactly like the vector path; only
            // the aligned input's final partial chunk (fewer than four items) ends the loop.
            _Pragma("unroll")
            for (uint32_t item = 0; item < 4U; ++item) {
                if (offset + item < input_size) {
                    process(input[offset + item]);
                }
            }
            if (vector_input) {
                break;
            }
        }
    }
    __syncthreads();
 
    // Each CTA starts its merge at a rotated bucket offset (odd multiplier, coprime with
    // the power-of-two bucket count), so the CTAs' atomic merges interleave across different
    // addresses instead of all colliding on the same bucket at once. The per-address atomic
    // serialization then sees a smooth stream of arrivals rather than a synchronized burst.
    auto const bucket_offset = (static_cast<size_t>(blockIdx.x) * 139U) & (BucketCount - 1U);
    for (auto j = threadIdx.x; j < BucketCount; j += blockDim.x) {
        auto const i = (static_cast<size_t>(j) + bucket_offset) & (BucketCount - 1U);
        auto const stored = state[i];
        if (stored != 0U) atomicMax(&registers[i], stored);
    }
}

block

auto const cuddl::detail::block = static_cast<uint64_t>(blockIdx.x) * (blockDim.x / 32U) + threadIdx.x / 32U

Definition at line 236 of file top_k.cuh.

block_size

constexpr uint32_t cuddl::detail::block_size = 256

constexpr

Threads per CTA for the single-pair and cardinality reduction kernels.

Definition at line 28 of file kernels.cuh.

cached_groups

constexpr uint32_t cuddl::detail::cached_groups = lane_groups < 2U ? lane_groups : 2U

constexpr

Definition at line 1866 of file kernels.cuh.

candidate_bits

__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t uint32_t const * cuddl::detail::candidate_bits

Initial value:

{
    auto const words = candidate_bit_words(reference_count)

Definition at line 929 of file kernels.cuh.

candidate_count

__global__ uint32_t const uint32_t uint32_t const uint32_t const* cuddl::detail::candidate_count

Definition at line 1542 of file kernels.cuh.

candidate_ids

__global__ uint32_t const uint32_t uint32_t const* cuddl::detail::candidate_ids

Definition at line 1541 of file kernels.cuh.

carry

uint32_t cuddl::detail::carry = 0U

Definition at line 547 of file kernels.cuh.

cell_counter

__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t uint32_t const SearchResult uint32_t uint32_t uint32_t uint32_t* cuddl::detail::cell_counter

Initial value:

{
    constexpr uint32_t warp_width = 32

Definition at line 1855 of file kernels.cuh.

cells

auto const cuddl::detail::cells

Initial value:

= make_refine_cells<UpperTriangle>(
        query_id_offset,
        query_count,
        group,
        words_per_query,
        max_block_words,
        Candidates == refine_candidates::bitmap ? bitmap_refine_waves : 1U
    )

Definition at line 1878 of file kernels.cuh.

claimed_cell

__shared__ uint32_t cuddl::detail::claimed_cell

Definition at line 1874 of file kernels.cuh.

column

uint32_t cuddl::detail::column = 0U

Definition at line 247 of file top_k.cuh.

columns

__global__ uint32_t* cuddl::detail::columns

Definition at line 234 of file top_k.cuh.

count

__global__ uint32_t cuddl::detail::count

Initial value:

= [&](uint32_t reference) {
        auto const slot = reference - low;
        atomicAdd(&packed_counts[slot / 2U], 1U << ((slot % 2U) * 16U));
    }

Definition at line 1031 of file kernels.cuh.

                                               {
        auto const slot = reference - low;
        atomicAdd(&packed_counts[slot / 2U], 1U << ((slot % 2U) * 16U));
    };

ddl_cf_size

constexpr uint32_t cuddl::detail::ddl_cf_size = 13

constexpr

Definition at line 25 of file detail/hybrid_cardinality.cuh.

default_arena_ceiling

constexpr size_t cuddl::detail::default_arena_ceiling = size_t{128} << 20

inlineconstexpr

Default arena ceiling.

A batch is sketched only once its arena fills, so an arena that holds the whole corpus leaves the device idle until the last genome loads. Bounding it lets each batch's kernel overlap the loading of the next: on a GH200 building 2048 genomes this cut the build from 239 to 194 ms, and it cost nothing on a 24-thread RTX 5070 Ti host.

Definition at line 117 of file database_staging.cuh.

{128} << 20;

delta_candidates

constexpr uint32_t cuddl::detail::delta_candidates = delta_signatures * delta_signature_candidates

constexpr

Definition at line 26 of file reference_deltas.cuh.

delta_change_limit

constexpr uint32_t cuddl::detail::delta_change_limit = 160U

constexpr

Most buckets in which a child reference may differ from its base reference.

Definition at line 20 of file reference_deltas.cuh.

delta_child_tile

constexpr uint32_t cuddl::detail::delta_child_tile = 2048U

constexpr

Definition at line 30 of file reference_deltas.cuh.

delta_correction_shared_bytes

template<size_t BucketCount>

constexpr size_t cuddl::detail::delta_correction_shared_bytes

constexpr

Initial value:

=
    BucketCount * delta_query_group<BucketCount> * sizeof(uint16_t) +
    size_t{delta_correction_warps} * delta_change_limit * sizeof(uint2)

Definition at line 39 of file reference_deltas.cuh.

          {delta_correction_warps} * delta_change_limit * sizeof(uint2);

delta_correction_warps

constexpr uint32_t cuddl::detail::delta_correction_warps = 24U

constexpr

Warps per apply_reference_deltas_kernel block, and the most and fewest children one block cell takes.

Definition at line 29 of file reference_deltas.cuh.

delta_min_child_tile

constexpr uint32_t cuddl::detail::delta_min_child_tile = 4U * delta_correction_warps

constexpr

Definition at line 31 of file reference_deltas.cuh.

delta_query_group

template<size_t BucketCount>

constexpr uint32_t cuddl::detail::delta_query_group = static_cast<uint32_t>(32768U / BucketCount)

constexpr

Queries one correction cell stages bucket-major: 64 KiB of scores for every bucket count.

Lane l serves query l % group and every (32 / group)-th change from l / group.

Definition at line 36 of file reference_deltas.cuh.

delta_signature_buckets

constexpr uint32_t cuddl::detail::delta_signature_buckets = 8U

constexpr

Definition at line 23 of file reference_deltas.cuh.

delta_signature_candidates

constexpr uint32_t cuddl::detail::delta_signature_candidates = 4U

constexpr

Highest-ID rows taken from each matching signature.

Definition at line 25 of file reference_deltas.cuh.

delta_signatures

constexpr uint32_t cuddl::detail::delta_signatures = 8U

constexpr

Exact bucket signatures that propose base candidates, and buckets per signature.

Definition at line 22 of file reference_deltas.cuh.

directory

__global__ size_t uint32_t uint16_t const uint32_t const * cuddl::detail::directory

Initial value:

{
    auto const entries = sparse_directory_entries(reference_count)

Definition at line 439 of file kernels.cuh.

dlc_blend_high

constexpr double cuddl::detail::dlc_blend_high = 0.952

constexpr

Definition at line 14 of file detail/hybrid_cardinality.cuh.

dlc_blend_low

constexpr double cuddl::detail::dlc_blend_low = 0.088

constexpr

Definition at line 13 of file detail/hybrid_cardinality.cuh.

dlc_info_power

constexpr double cuddl::detail::dlc_info_power = 4.5

constexpr

Definition at line 16 of file detail/hybrid_cardinality.cuh.

dlc_min_fraction

constexpr double cuddl::detail::dlc_min_fraction = 0.002

constexpr

Definition at line 15 of file detail/hybrid_cardinality.cuh.

else

cuddl::detail::else

Initial value:

{
        for (auto i = threadIdx.x; i < (width + 1U) / 2U; i += blockDim.x) {
            packed_counts[i] = 0U;
        }
    }
    if (threadIdx.x == 0U) {
        long_count = 0U;
    }
    __syncthreads()

Definition at line 1020 of file kernels.cuh.

           {
        for (auto i = threadIdx.x; i < (width + 1U) / 2U; i += blockDim.x) {
            packed_counts[i] = 0U;
        }
    }
    if (threadIdx.x == 0U) {
        long_count = 0U;
    }

first_query_id

__global__ uint32_t const uint32_t const uint32_t const uint32_t cuddl::detail::first_query_id

Definition at line 1786 of file kernels.cuh.

first_word

__global__ uint32_t uint32_t cuddl::detail::first_word

Initial value:

{
    auto const lane = static_cast<uint32_t>(threadIdx.x) % 32U

Definition at line 234 of file top_k.cuh.

group

constexpr uint32_t cuddl::detail::group = QueryGroup

constexpr

Definition at line 1858 of file kernels.cuh.

groups

auto const cuddl::detail::groups = cells.groups

Definition at line 1886 of file kernels.cuh.

groups_per_row

constexpr uint32_t cuddl::detail::groups_per_row = static_cast<uint32_t>(BucketCount / warp_width)

constexpr

Definition at line 1360 of file kernels.cuh.

hash_range

constexpr double cuddl::detail::hash_range = 0x1p64

constexpr

2^64, the range of the 64-bit hash used by the sketch.

Definition at line 10 of file cardinality.cuh.

heap

auto* const cuddl::detail::heap = heaps + warp * top_k_heap_words(slots.k)

Definition at line 200 of file top_k.cuh.

high

auto const cuddl::detail::high = min(low + tile, reference_count)

Definition at line 940 of file kernels.cuh.

hybrid_high

constexpr double cuddl::detail::hybrid_high = 5.0

constexpr

Definition at line 18 of file detail/hybrid_cardinality.cuh.

hybrid_low

constexpr double cuddl::detail::hybrid_low = 0.2

constexpr

Definition at line 17 of file detail/hybrid_cardinality.cuh.

ids

__global__ uint32_t uint32_t uint64_t uint32_t* cuddl::detail::ids

Initial value:

{
    constexpr uint32_t spacing =
        static_cast<uint32_t>(BucketCount / (delta_signatures * delta_signature_buckets))

Definition at line 85 of file reference_deltas.cuh.

IncludeCardinality

__global__ size_t uint32_t bool blacklist_view bool cuddl::detail::IncludeCardinality

Definition at line 155 of file kernels.cuh.

index_match_cells_per_warp

constexpr uint32_t cuddl::detail::index_match_cells_per_warp = 32U

constexpr

Cells one warp owns per iteration of count_batch_index_matches_kernel: one per lane.

The host divides the grid by this so every launched warp stays busy.

Definition at line 392 of file kernels.cuh.

index_pair_fraction_limit

constexpr double cuddl::detail::index_pair_fraction_limit = 0.03

constexpr

Index pair fraction (reference_index::pair_fraction) above which batch search compares bit-planes for every pair instead of counting through the index.

Definition at line 952 of file reference_database.cuh.

index_tile_lane_postings

constexpr uint32_t cuddl::detail::index_tile_lane_postings = 16U

constexpr

Longest posting range one count_batch_index_tile_kernel lane walks alone.

Definition at line 682 of file kernels.cuh.

index_tile_long_capacity

constexpr uint32_t cuddl::detail::index_tile_long_capacity = 32U

constexpr

Long posting ranges one count_batch_index_tile_kernel block walks together.

Definition at line 685 of file kernels.cuh.

indexed_bucket_count

__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t cuddl::detail::indexed_bucket_count

Definition at line 438 of file kernels.cuh.

indexed_groups

auto const cuddl::detail::indexed_groups = indexed_bucket_count / warp_width

Definition at line 1594 of file kernels.cuh.

input_size

__global__ size_t cuddl::detail::input_size

Definition at line 93 of file kernels.cuh.

items

auto const cuddl::detail::items = candidate_ids == nullptr ? reference_count : *candidate_count

Definition at line 1549 of file kernels.cuh.

key_count

auto const cuddl::detail::key_count = static_cast<uint32_t>(key_mask) + 1U

Definition at line 472 of file kernels.cuh.

key_mask

__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t cuddl::detail::key_mask

Definition at line 540 of file kernels.cuh.

keys

__global__ uint32_t uint32_t uint64_t* cuddl::detail::keys

Definition at line 84 of file reference_deltas.cuh.

lane

auto const cuddl::detail::lane = static_cast<uint32_t>(threadIdx.x) % warp_width

Definition at line 966 of file kernels.cuh.

lane_groups

constexpr uint32_t cuddl::detail::lane_groups = groups_per_row / warp_width

constexpr

Definition at line 1864 of file kernels.cuh.

Layout

__global__ size_t uint32_t bool blacklist_view bool cuddl::detail::Layout

Definition at line 155 of file kernels.cuh.

                                                                                      {
    using block_reduce = cub::BlockReduce<summary_payload, block_size>;
    __shared__ typename block_reduce::TempStorage storage;
 
    summary_payload local{};
    for (auto bucket = static_cast<size_t>(threadIdx.x); bucket < BucketCount;
         bucket += block_size) {
        auto const left_reg = left[bucket];
        auto const right_reg = right[bucket];
        classify(local.counts, left_reg, right_reg);
        if constexpr (IncludeCardinality) {
            if (left_reg == 0U) {
                ++local.empty;
            } else {
                local.restored_sum += restore_midpoint<Layout>(static_cast<uint16_t>(left_reg));
            }
        }
    }
    auto const total = block_reduce(storage).Reduce(local, combine_payloads<IncludeCardinality>);
    if (threadIdx.x == 0) {
        output.counts = total.counts;
        if constexpr (IncludeCardinality) {
            output.cardinality = cardinality(
                static_cast<double>(BucketCount),
                static_cast<double>(total.empty),
                total.restored_sum
            );
        }
    }
}
 
template <size_t BucketCount>
__global__ __launch_bounds__(block_size) void batch_summary_kernel(
    uint32_t const* left_rows,
    uint32_t const* right_rows,
    size_t pair_count,
    pairwise_summary* outputs
) {
    constexpr uint32_t warp_width = 32U;
    constexpr uint32_t warps_per_block = block_size / warp_width;
    using warp_reduce = cub::WarpReduce<pairwise_counts>;
    __shared__ typename warp_reduce::TempStorage storage[warps_per_block];
 
    auto const warp = static_cast<uint32_t>(threadIdx.x) / warp_width;
    auto const lane = static_cast<uint32_t>(threadIdx.x) % warp_width;
    auto pair = static_cast<size_t>(blockIdx.x) * warps_per_block + warp;
    auto const pair_stride = static_cast<size_t>(gridDim.x) * warps_per_block;
    for (; pair < pair_count; pair += pair_stride) {
        auto const row_offset = pair * BucketCount;
        pairwise_counts local{};
        for (auto bucket = static_cast<size_t>(lane); bucket < BucketCount; bucket += warp_width) {
            classify(local, left_rows[row_offset + bucket], right_rows[row_offset + bucket]);
        }
        auto const total = warp_reduce(storage[warp]).Sum(local);
        if (lane == 0U) {
            outputs[pair].counts = total;
            outputs[pair].cardinality = 0.0;
        }
        __syncwarp();
    }
}
 
inline __device__ pairwise_counts
reduce_warp(cg::thread_block_tile<32> const warp, pairwise_counts counts) {
    counts.lower = cg::reduce(warp, counts.lower, cg::plus<uint32_t>{});
    counts.equal = cg::reduce(warp, counts.equal, cg::plus<uint32_t>{});
    counts.higher = cg::reduce(warp, counts.higher, cg::plus<uint32_t>{});
    counts.both_empty = cg::reduce(warp, counts.both_empty, cg::plus<uint32_t>{});
    return counts;
}
 
template <typename Row>
__global__ void transpose_indexed_scores_kernel(
    Row const* rows,
    uint32_t reference_count,
    uint32_t indexed_bucket_count,
    uint32_t full_bucket_count,
    Row* transposed
) {
    constexpr uint32_t tile = 32U;
    __shared__ Row staging[tile][tile + 1U];
    auto const bucket = static_cast<uint32_t>(blockIdx.x) * tile + threadIdx.x;
    auto const reference = static_cast<uint32_t>(blockIdx.y) * tile + threadIdx.y;
    if (bucket < indexed_bucket_count && reference < reference_count) {
        staging[threadIdx.y][threadIdx.x] =
            rows[static_cast<size_t>(reference) * full_bucket_count + bucket];
    }
    __syncthreads();
    // Output (bucket, reference) = input (reference, bucket). Threads with consecutive
    // threadIdx.x must write consecutive references of one bucket, so the output bucket index
    // comes from threadIdx.y and the output reference from threadIdx.x; the shared-memory read
    // is the transposed (column) access instead.
    auto const out_bucket = static_cast<uint32_t>(blockIdx.x) * tile + threadIdx.y;
    auto const out_reference = static_cast<uint32_t>(blockIdx.y) * tile + threadIdx.x;
    if (out_bucket < indexed_bucket_count && out_reference < reference_count) {
        transposed[static_cast<size_t>(out_bucket) * reference_count + out_reference] =
            staging[threadIdx.x][threadIdx.y];
    }
}
 
template <typename Row>
__global__ void count_index_cells_bucket_kernel(
    Row const* transposed,
    uint32_t indexed_bucket_count,
    uint32_t reference_count,
    uint16_t key_mask,
    uint32_t* cell_counts
) {
    extern __shared__ uint32_t key_counts[];
    auto const key_count = static_cast<uint32_t>(key_mask) + 1U;
    auto const quarter_keys = key_count / 4U;
    // The exclusive scan also reads the trailing element used for the CSR end offset.
    if (blockIdx.x == 0U && threadIdx.x == 0U) {
        cell_counts[static_cast<size_t>(indexed_bucket_count) * key_count] = 0U;
    }
    for (auto i = threadIdx.x; i < quarter_keys; i += blockDim.x) {
        key_counts[i] = 0U;
    }
    __syncthreads();
 
    for (auto bucket = blockIdx.x; bucket < indexed_bucket_count; bucket += gridDim.x) {
        auto const* bucket_scores = transposed + static_cast<size_t>(bucket) * reference_count;
        auto* bucket_counts = cell_counts + static_cast<size_t>(bucket) * key_count;
        for (uint32_t quarter = 0U; quarter < 4U; ++quarter) {
            auto const key_base = quarter * quarter_keys;
            for (uint32_t reference = threadIdx.x; reference < reference_count;
                 reference += blockDim.x) {
                auto const score = static_cast<uint16_t>(__ldcs(&bucket_scores[reference]));
                if (score == 0U) {
                    continue;
                }
                auto const key = static_cast<uint32_t>(score & key_mask);
                if (key < key_base || key >= key_base + quarter_keys) {
                    continue;
                }
                atomicAdd(&key_counts[key - key_base], 1U);
            }
            __syncthreads();
            for (auto i = threadIdx.x; i < quarter_keys; i += blockDim.x) {
                bucket_counts[key_base + i] = key_counts[i];
                key_counts[i] = 0U;
            }
            __syncthreads();
        }
    }
}
 
template <typename Row>
__global__ void scatter_index_postings_bucket_kernel(
    Row const* transposed,
    uint32_t indexed_bucket_count,
    uint32_t reference_count,
    uint16_t key_mask,
    uint32_t const* offsets,
    uint32_t* postings
) {
    extern __shared__ uint32_t key_ranks[];
    auto const key_count = static_cast<uint32_t>(key_mask) + 1U;
    auto const quarter_keys = key_count / 4U;
    for (auto i = threadIdx.x; i < quarter_keys; i += blockDim.x) {
        key_ranks[i] = 0U;
    }
    __syncthreads();
 
    for (auto bucket = blockIdx.x; bucket < indexed_bucket_count; bucket += gridDim.x) {
        auto const* bucket_scores = transposed + static_cast<size_t>(bucket) * reference_count;
        auto const* bucket_offsets = offsets + static_cast<size_t>(bucket) * key_count;
        for (uint32_t quarter = 0U; quarter < 4U; ++quarter) {
            auto const key_base = quarter * quarter_keys;
            for (uint32_t reference = threadIdx.x; reference < reference_count;
                 reference += blockDim.x) {
                auto const score = static_cast<uint16_t>(__ldcs(&bucket_scores[reference]));
                if (score == 0U) {
                    continue;
                }
                auto const key = static_cast<uint32_t>(score & key_mask);
                if (key < key_base || key >= key_base + quarter_keys) {
                    continue;
                }
                auto const rank = atomicAdd(&key_ranks[key - key_base], 1U);
                __stcs(&postings[bucket_offsets[key] + rank], reference);
            }
            __syncthreads();
            for (auto i = threadIdx.x; i < quarter_keys; i += blockDim.x) {
                key_ranks[i] = 0U;
            }
            __syncthreads();
        }
    }
}
 
struct minimum_match_predicate {
    uint32_t const* match_counts;
    uint32_t minimum_matches;
 
    [[nodiscard]] __host__ __device__ bool operator()(uint32_t reference_id) const noexcept {
        return match_counts[reference_id] >= minimum_matches;
    }
};

local

unsigned long long cuddl::detail::local = 0U

Definition at line 478 of file kernels.cuh.

long_count

__shared__ uint32_t cuddl::detail::long_count

Definition at line 965 of file kernels.cuh.

long_ranges

__shared__ uint2 cuddl::detail::long_ranges[index_tile_long_capacity]

Definition at line 964 of file kernels.cuh.

long_ranges_used

auto const cuddl::detail::long_ranges_used

Initial value:

=
        long_count < index_tile_long_capacity ? long_count : index_tile_long_capacity

Definition at line 1152 of file kernels.cuh.

low

auto cuddl::detail::low = static_cast<uint32_t>(blockIdx.x) % tiles * tile

Definition at line 939 of file kernels.cuh.

mantissa_bits

constexpr uint8_t cuddl::detail::mantissa_bits = default_register_layout::mantissa_bits

constexpr

Default score layout retained for source compatibility.

Definition at line 27 of file register.cuh.

mantissa_mask

constexpr uint32_t cuddl::detail::mantissa_mask = (1U << mantissa_bits) - 1U

constexpr

Definition at line 28 of file register.cuh.

match_counts

__global__ uint32_t const * cuddl::detail::match_counts

Initial value:

{
    auto const total = range_end_scan[indexed_bucket_count - 1U]

Definition at line 575 of file kernels.cuh.

match_filter_block_size

constexpr uint32_t cuddl::detail::match_filter_block_size = 512U

constexpr

Threads per match_filter_kernel block.

Four references' low planes stay in registers, so a block holds fewer threads than a refine block.

Definition at line 2090 of file kernels.cuh.

match_filter_planes

constexpr uint32_t cuddl::detail::match_filter_planes = 12U

constexpr

Low planes match_filter_kernel compares.

Definition at line 2110 of file kernels.cuh.

match_filter_supported

template<size_t BucketCount>

constexpr bool cuddl::detail::match_filter_supported

constexpr

Initial value:

=
    bitmap_refine_staged<BucketCount> && BucketCount % 1024U == 0U && BucketCount <= 2048U

Whether match_filter_kernel supports BucketCount: a lane caches at most two plane groups of each of its references.

Definition at line 2095 of file kernels.cuh.

max_block_words

constexpr uint32_t cuddl::detail::max_block_words

constexpr

Initial value:

= static_cast<uint32_t>(cuda::std::max<size_t>(
        1U, refine_block_row_bytes / (BucketCount * sizeof(uint16_t)) / warp_width
    ))

Definition at line 1867 of file kernels.cuh.

max_nlz

constexpr uint32_t cuddl::detail::max_nlz = restore_shift_base

constexpr

Definition at line 32 of file register.cuh.

maximum_batch_rows

constexpr size_t cuddl::detail::maximum_batch_rows = (size_t{1} << 31) - 1U

constexpr

Stored sketches one batch launch may index, bounded by the CUDA grid dimension.

Definition at line 31 of file kernels.cuh.

minimum_matches

__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t uint32_t const SearchResult uint32_t uint32_t cuddl::detail::minimum_matches

Definition at line 926 of file kernels.cuh.

missing_cell

constexpr uint32_t cuddl::detail::missing_cell = 0x00FFFFFFU

inlineconstexpr

Eight ambiguous bases: an encoder cell past the end of a sequence.

Definition at line 27 of file dna.hpp.

next_word

__shared__ uint32_t cuddl::detail::next_word

Definition at line 1873 of file kernels.cuh.

nlz_bins

constexpr int32_t cuddl::detail::nlz_bins = 66

constexpr

NLZ histogram bins for the hybrid estimator, including the empty tier.

Definition at line 12 of file detail/hybrid_cardinality.cuh.

offsets

__global__ size_t uint32_t const * cuddl::detail::offsets

Definition at line 536 of file kernels.cuh.

packed

__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t uint32_t SearchResult* cuddl::detail::packed

Definition at line 1791 of file kernels.cuh.

packed_base

__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t uint32_t cuddl::detail::packed_base

Definition at line 1790 of file kernels.cuh.

packed_buffer_bytes

constexpr size_t cuddl::detail::packed_buffer_bytes = size_t{2} << 20

inlineconstexpr

Bytes reserved per loader buffer that holds packed bases: a bacterial genome's two-bit words, about 1.5 MB, fit.

Definition at line 208 of file database_staging.cuh.

{2} << 20;

packed_counts

__shared__ uint32_t cuddl::detail::packed_counts[]

extern

packed_match_counts

__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t uint32_t SearchResult uint32_t* cuddl::detail::packed_match_counts

Initial value:

{
    constexpr uint32_t warp_width = 32

Definition at line 1792 of file kernels.cuh.

pairs

auto const cuddl::detail::pairs = static_cast<uint64_t>(query_count) * reference_count

Definition at line 1592 of file kernels.cuh.

parallel_read_depth

constexpr size_t cuddl::detail::parallel_read_depth = 64

inlineconstexpr

Reads a parallel file system needs in flight to give one node its bandwidth: on MOGON's Lustre, 16 threads read 2.9 GB/s of ~1 MB files and 64 threads 6.7 GB/s.

Definition at line 1784 of file fastx_sequence_file.hpp.

pass_bits

__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t uint32_t const SearchResult uint32_t uint32_t uint32_t * cuddl::detail::pass_bits

Definition at line 1784 of file kernels.cuh.

planes

__global__ uint32_t uint32_t* cuddl::detail::planes

Initial value:

{
    constexpr uint32_t warp_width = 32

Definition at line 1358 of file kernels.cuh.

posting_range_block_size

constexpr uint32_t cuddl::detail::posting_range_block_size = 1024U

constexpr

Threads of the single index_posting_ranges_kernel block.

Definition at line 527 of file kernels.cuh.

postings

__global__ size_t uint32_t const uint32_t const * cuddl::detail::postings

Definition at line 574 of file kernels.cuh.

PRIME64_1

constexpr uint64_t cuddl::detail::PRIME64_1 = 11400714785074694791ULL

constexpr

xxHash-64 round primes.

Definition at line 33 of file hash.cuh.

PRIME64_2

constexpr uint64_t cuddl::detail::PRIME64_2 = 14029467366897019727ULL

constexpr

Definition at line 34 of file hash.cuh.

PRIME64_3

constexpr uint64_t cuddl::detail::PRIME64_3 = 1609587929392839161ULL

constexpr

Definition at line 35 of file hash.cuh.

PRIME64_4

constexpr uint64_t cuddl::detail::PRIME64_4 = 9650029242287828579ULL

constexpr

Definition at line 36 of file hash.cuh.

PRIME64_5

constexpr uint64_t cuddl::detail::PRIME64_5 = 2870177450012600261ULL

constexpr

Definition at line 37 of file hash.cuh.

query

auto const cuddl::detail::query = queries + (query_row_offset + query_index) * BucketCount

Definition at line 1030 of file kernels.cuh.

query_count

__global__ uint32_t uint32_t cuddl::detail::query_count

Definition at line 600 of file kernels.cuh.

query_id

auto const cuddl::detail::query_id = tile.first_query_id + query

Definition at line 201 of file top_k.cuh.

query_id_offset

__global__ uint32_t cuddl::detail::query_id_offset

Definition at line 927 of file kernels.cuh.

query_index

auto const cuddl::detail::query_index = static_cast<uint32_t>(blockIdx.x) / tiles

Definition at line 938 of file kernels.cuh.

query_plane_storage

__shared__ uint4 cuddl::detail::query_plane_storage[]

extern

query_row_offset

__global__ size_t cuddl::detail::query_row_offset

Definition at line 599 of file kernels.cuh.

range_begin

__global__ uint32_t const uint16_t const uint32_t uint32_t uint16_t uint32_t* cuddl::detail::range_begin

Definition at line 541 of file kernels.cuh.

range_end_scan

static __global__ uint32_t const * cuddl::detail::range_end_scan

Initial value:

{
    using block_scan = cub::BlockScan<uint32_t, posting_range_block_size>

Definition at line 542 of file kernels.cuh.

ranges

__global__ size_t uint32_t uint16_t const uint32_t const uint32_t uint32_t uint16_t uint2 * cuddl::detail::ranges

Initial value:

{
    auto const bucket = static_cast<uint32_t>(blockIdx.x)

Definition at line 925 of file kernels.cuh.

reference_count

__global__ uint32_t uint32_t uint32_t cuddl::detail::reference_count

Definition at line 437 of file kernels.cuh.

reference_mask

__global__ uint32_t uint32_t uint32_t SearchResult uint32_t const* cuddl::detail::reference_mask

Initial value:

{
    constexpr uint32_t warp_width = 32

Definition at line 2491 of file kernels.cuh.

reference_planes

__global__ uint32_t uint32_t uint32_t const * cuddl::detail::reference_planes

Definition at line 1539 of file kernels.cuh.

refine_block_row_bytes

constexpr size_t cuddl::detail::refine_block_row_bytes = size_t{8} << 20U

constexpr

Reference-row bytes one refine_batch_index_candidates_kernel block spans.

Candidates are refined one (reference block, query) cell at a time, reference-block major, so the resident warps share a few blocks' rows through L2 instead of every query streaming every candidate reference row from DRAM.

Definition at line 1323 of file kernels.cuh.

{8} << 20U;

registers

__global__ size_t uint32_t* cuddl::detail::registers

Definition at line 94 of file kernels.cuh.

restore_shift_base

constexpr uint32_t cuddl::detail::restore_shift_base = 63U - mantissa_bits

constexpr

Default restoration constants retained for source compatibility.

Definition at line 31 of file register.cuh.

result_match_counts

__global__ uint32_t uint32_t uint32_t const uint32_t uint32_t uint16_t uint32_t const SearchResult uint32_t* cuddl::detail::result_match_counts

Definition at line 1852 of file kernels.cuh.

results

__global__ uint32_t uint32_t uint32_t SearchResult * cuddl::detail::results

Initial value:

{
    constexpr uint32_t warp_width = 32

Definition at line 1543 of file kernels.cuh.

row

auto const cuddl::detail::row = static_cast<size_t>(query_id - slots.first_genome_id)

Definition at line 202 of file top_k.cuh.

row_count

__global__ size_t cuddl::detail::row_count

Definition at line 1358 of file kernels.cuh.

row_word

auto const cuddl::detail::row_word = static_cast<uint32_t>(block / word_count)

Definition at line 242 of file top_k.cuh.

row_words

auto const cuddl::detail::row_words = static_cast<uint32_t>(BucketCount / 2U)

constexpr

Definition at line 1546 of file kernels.cuh.

rows

__global__ size_t uint16_t* cuddl::detail::rows

Initial value:

{
    constexpr uint32_t warp_width = 32

Definition at line 1499 of file kernels.cuh.

score_planes

constexpr uint32_t cuddl::detail::score_planes = 16U

constexpr

Bits per score, and so bit-planes per 32-bucket group.

Definition at line 1326 of file kernels.cuh.

seed

constexpr uint64_t cuddl::detail::seed = 42

constexpr

Domain separation constant folded into every packed k-mer before hashing.

Definition at line 18 of file hash.cuh.

shared_construction_block_size

constexpr uint32_t cuddl::detail::shared_construction_block_size = 768

constexpr

Threads per CTA for the CTA-local construction kernel.

Definition at line 78 of file kernels.cuh.

shift

auto const cuddl::detail::shift = sparse_directory_shift(reference_count)

Definition at line 442 of file kernels.cuh.

signature

__global__ uint32_t uint32_t cuddl::detail::signature

Definition at line 83 of file reference_deltas.cuh.

skipped

auto const cuddl::detail::skipped = UpperTriangle ? (query_id + 1U) / 32U : 0U

Definition at line 206 of file top_k.cuh.

slots

__global__ top_k_slots cuddl::detail::slots

Initial value:

{
    extern __shared__ uint64_t heaps[]

Definition at line 195 of file top_k.cuh.

sorted_keys

__global__ size_t uint32_t uint16_t const * cuddl::detail::sorted_keys

Definition at line 469 of file kernels.cuh.

staged

constexpr bool cuddl::detail::staged = bitmap_refine_staged<BucketCount>

constexpr

Definition at line 1860 of file kernels.cuh.

stager_arenas

constexpr size_t cuddl::detail::stager_arenas = 4

inlineconstexpr

Staging arenas the copy stream rotates through, so copies run ahead of the kernels that read earlier arenas while other work shares the device.

Definition at line 121 of file database_staging.cuh.

stager_hold_slots

constexpr size_t cuddl::detail::stager_hold_slots = 16

inlineconstexpr

Host memory the device still has to read must stay alive: the transfer engine's read of a pageable buffer is not ordered with the host writes that follow it.

Sixteen slots take the tail off that wait; at four the tail reached hundreds of milliseconds while dozens of loaders fed one consumer.

Definition at line 204 of file database_staging.cuh.

storage

__shared__ block_scan::TempStorage cuddl::detail::storage

Definition at line 545 of file kernels.cuh.

stride

auto const cuddl::detail::stride = static_cast<uint32_t>(gridDim.x) * block_size

Definition at line 578 of file kernels.cuh.

thresholded

auto const cuddl::detail::thresholded = CountMatches && pass_bits != nullptr

Definition at line 1888 of file kernels.cuh.

tile

__global__ size_t uint32_t const uint32_t const uint32_t uint32_t uint16_t uint32_t cuddl::detail::tile

Definition at line 922 of file kernels.cuh.

tiles

auto const cuddl::detail::tiles = (reference_count + tile - 1U) / tile

Definition at line 937 of file kernels.cuh.

top_k_warps

constexpr uint32_t cuddl::detail::top_k_warps = block_size / 32U

constexpr

Warps per merging block, each holding one heap; fewer when the heaps exceed shared memory.

Definition at line 53 of file top_k.cuh.

total

auto const cuddl::detail::total = static_cast<size_t>(indexed_bucket_count) * entries

Definition at line 443 of file kernels.cuh.

vector_input

__global__ size_t uint32_t bool cuddl::detail::vector_input

Definition at line 95 of file kernels.cuh.

warp

auto const cuddl::detail::warp = static_cast<uint32_t>(threadIdx.x) / warp_width

Definition at line 967 of file kernels.cuh.

warps

auto const cuddl::detail::warps = uint64_t{gridDim.x} * (block_size / warp_width)

Definition at line 2495 of file kernels.cuh.

{gridDim.x} * (block_size / warp_width);

warps_per_block

constexpr uint32_t cuddl::detail::warps_per_block = index_tile_block_size / warp_width

constexpr

Definition at line 935 of file kernels.cuh.

wide_chunk_buckets

template<typename QueryScore , typename ReferenceScore >

constexpr uint32_t cuddl::detail::wide_chunk_buckets

constexpr

Initial value:

=
    (sizeof(QueryScore) == 2U || sizeof(ReferenceScore) == 2U) ? 16U : 8U

Buckets one contiguous chunk covers, for a query/reference score pair.

One wide load covers 16 buckets whenever either row stores 16-bit scores, and 8 buckets when both store 32-bit registers. The scalar fallback covers half as many.

Definition at line 74 of file kernels.cuh.

width

auto const cuddl::detail::width = high - low

Definition at line 963 of file kernels.cuh.

word

auto const cuddl::detail::word = first_word + static_cast<uint32_t>(block % word_count)

Definition at line 241 of file top_k.cuh.

word_begin

__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t cuddl::detail::word_begin

Definition at line 1788 of file kernels.cuh.

word_count

auto const cuddl::detail::word_count = words - first_word

Definition at line 238 of file top_k.cuh.

word_end

__global__ uint32_t const uint32_t const uint32_t const uint32_t uint32_t size_t size_t cuddl::detail::word_end

Definition at line 1789 of file kernels.cuh.

word_offsets

__global__ uint32_t const uint32_t const uint32_t const* cuddl::detail::word_offsets

Definition at line 1785 of file kernels.cuh.

word_slices

constexpr uint32_t cuddl::detail::word_slices = Candidates == refine_candidates::all ? 1U : 4U

constexpr

Definition at line 1871 of file kernels.cuh.

words

auto const cuddl::detail::words = tile.words_per_query()

Definition at line 204 of file top_k.cuh.

words_per_query

auto const cuddl::detail::words_per_query = candidate_bit_words(reference_count)

Definition at line 1795 of file kernels.cuh.

work

__global__ uint16_t const uint32_t uint32_t uint32_t unsigned long long* cuddl::detail::work

Initial value:

{
    auto const total = offsets != nullptr
                           ? static_cast<size_t>(indexed_bucket_count) * key_count
                           : static_cast<size_t>(indexed_bucket_count) * reference_count

Definition at line 473 of file kernels.cuh.