cuDDL
Loading...
Searching...
No Matches
Classes | Public Member Functions | Static Public Member Functions | Static Public Attributes | List of all members
cuddl::detail::device_fasta_pipeline Class Reference

Normalises FASTA batches supplied compressed, uncompressed, or already inflated. More...

Public Member Functions

 device_fasta_pipeline (cuda::stream_ref stream, size_t slot_capacity, size_t compressed_capacity, size_t workers, bool hardware_inflate)
 
size_t slot_capacity () const noexcept
 
size_t compressed_capacity () const noexcept
 
size_t hardware_limit () const noexcept
 Largest inflated file the device's decompression engine takes; 0 without one.
 
size_t batch_files (std::span< size_t const > ids, std::span< gzip_file_probe const > probes) const noexcept
 How many ids from the front of ids one lane holds.
 
Result< size_t > submit (size_t lane_index, std::span< size_t const > ids, std::span< std::filesystem::path const > paths, std::span< gzip_file_probe const > probes, bool direct_reads)
 Starts inflating files from the front of ids on lane.
 
Result< void > submit_decoded (size_t lane_index, std::span< decoded_fasta const > sources)
 Copies CPU-inflated FASTA into a lane and enqueues GPU normalisation.
 
Result< void > wait_lanes (cuda::stream_ref consumer)
 Completes lane work before callers release outstanding input buffers.
 
Result< void > finish (size_t lane_index, std::span< gzip_file_probe const > probes, std::vector< inflated_genome > &genomes, std::vector< size_t > &fallback)
 Waits for lane's batch and splits it into genomes the device holds and ids the host loader must take.
 
Result< void > release (size_t lane_index, cuda::stream_ref consumer)
 Lets lane reuse its slots once consumer reaches this point.
 

Static Public Member Functions

static size_t lane_overhead (size_t slot_capacity) noexcept
 Device bytes a lane needs besides its slots and compressed bytes.
 

Static Public Attributes

static constexpr size_t lane_count = 6
 
static constexpr size_t max_files = 4096
 
static constexpr size_t max_members = 65536
 Gzip members one batch inflates: one per file, or one per non-empty BGZF block.
 

Detailed Description

Normalises FASTA batches supplied compressed, uncompressed, or already inflated.

Each lane receives raw bytes through a batched copy, or reads files itself and inflates the gzip ones through nvCOMP. Kernels blank header text and leading bytes, then compaction drops whitespace. GPU-inflated data reuses compressed-input storage. CPU-inflated data compacts in place. One genome comes out as one run of bases with its records joined by the '>' that opened each one, a byte no k-mer window accepts, so it sketches exactly as its separate records would.

A file is handed back to the host loader when anything is off: an inflate failure, an output whose size or CRC32 disagrees with the trailer (a multi-member stream inflates only its first member here), FASTQ content, or no header. The host loader then gives it its usual result or error.

Definition at line 711 of file device_gzip.cuh.

Constructor & Destructor Documentation

◆ device_fasta_pipeline()

cuddl::detail::device_fasta_pipeline::device_fasta_pipeline ( cuda::stream_ref  stream,
size_t  slot_capacity,
size_t  compressed_capacity,
size_t  workers,
bool  hardware_inflate 
)
inline
Parameters
slot_capacityInflated bytes one lane holds, one guard byte per file included.
compressed_capacityCompressed bytes per lane; zero selects decoded input.
hardware_inflateWhether nvCOMP may use the device's decompression engine; without it, nvCOMP inflates on the SMs from ordinary device memory.

Definition at line 725 of file device_gzip.cuh.

732 : workers_(std::max<size_t>(1, workers)),
733 compressed_capacity_(compressed_capacity),
734 slot_capacity_(slot_capacity),
735 header_capacity_(
736 static_cast<uint32_t>(std::clamp<size_t>(
737 slot_capacity / 4096,
738 1 << 16,
739 std::numeric_limits<uint32_t>::max()
740 ))
741 ) {
742 auto const options = nvcompBatchedDeflateDecompressDefaultOpts;
743 nvcomp_temp_bytes_ = 0;
744 if (compressed_capacity_ != 0 &&
745 nvcompBatchedDeflateDecompressGetTempSizeAsync(
747 std::min<size_t>(slot_capacity_, std::numeric_limits<uint32_t>::max()),
748 options,
749 &nvcomp_temp_bytes_,
750 slot_capacity_
751 ) != nvcompSuccess) {
752 throw std::bad_alloc();
753 }
754 select_temp_bytes_ = 0;
755 auto const selected = compressed_capacity_ == 0 ? cub::DeviceSelect::If(
756 nullptr,
757 select_temp_bytes_,
758 static_cast<char*>(nullptr),
759 static_cast<int64_t*>(nullptr),
760 static_cast<int64_t>(slot_capacity_),
761 device_gzip::kept_byte{},
762 stream.get()
763 )
764 : cub::DeviceSelect::If(
765 nullptr,
766 select_temp_bytes_,
767 static_cast<char*>(nullptr),
768 static_cast<char*>(nullptr),
769 static_cast<int64_t*>(nullptr),
770 static_cast<int64_t>(slot_capacity_),
771 device_gzip::kept_byte{},
772 stream.get()
773 );
774 if (selected != cudaSuccess) throw std::runtime_error(cudaGetErrorString(selected));
775 auto const device = stream.device();
776 if (hardware_inflate) {
777 batch_decompress_ = device_gzip::batch_decompress_entry();
778 if (batch_decompress_ != nullptr) {
779 hardware_limit_ = device_gzip::hardware_inflate_limit(device.get());
780 }
781 }
782 streaming_window_ =
783 compressed_capacity_ == 0
784 ? std::min(
785 slot_capacity_,
786 static_cast<size_t>(
787 device.attribute(cuda::device_attributes::max_access_policy_window_size)
788 )
789 )
790 : size_t{0};
791 for (auto& lane : lanes_) {
792 lane.stream.emplace(device);
793 auto const s = cuda::stream_ref{*lane.stream};
794 lane.uploaded.emplace(s);
795 lane.classified.emplace(s);
796 lane.released.emplace(stream);
797 }
798 // The other lanes allocate on first use, so the device starts on the first batch while
799 // they page-lock their compressed buffers.
800 allocate(lanes_[0]);
801 }
static constexpr size_t max_members
Gzip members one batch inflates: one per file, or one per non-empty BGZF block.
size_t compressed_capacity() const noexcept
size_t slot_capacity() const noexcept
size_t hardware_inflate_limit(int device)
Largest output, in bytes, that device's hardware decompression engine (Blackwell B200,...
PFN_cuMemBatchDecompressAsync_v12060 batch_decompress_entry()
The driver's cuMemBatchDecompressAsync, or nullptr where the driver lacks it.
auto const lane
Definition kernels.cuh:966

Member Function Documentation

◆ batch_files()

size_t cuddl::detail::device_fasta_pipeline::batch_files ( std::span< size_t const >  ids,
std::span< gzip_file_probe const >  probes 
) const
inlinenoexcept

How many ids from the front of ids one lane holds.

Definition at line 822 of file device_gzip.cuh.

825 {
826 size_t files = 0, compressed = 0, slots = 0, members = 0;
827 while (files < ids.size() && files < max_files) {
828 auto const& probe = probes[ids[files]];
829 if (compressed + probe.host_bytes() > compressed_capacity_ ||
830 slots + probe.isize + 1 > slot_capacity_ || members + probe.members > max_members) {
831 break;
832 }
833 compressed += probe.host_bytes();
834 slots += probe.isize + 1;
835 members += probe.members;
836 ++files;
837 }
838 return files;
839 }
static constexpr size_t max_files
__global__ uint32_t uint32_t uint64_t uint32_t * ids
__global__ top_k_slots slots
Definition top_k.cuh:195

◆ compressed_capacity()

size_t cuddl::detail::device_fasta_pipeline::compressed_capacity ( ) const
inlinenoexcept

Definition at line 807 of file device_gzip.cuh.

807 {
808 return compressed_capacity_;
809 }

◆ finish()

Result< void > cuddl::detail::device_fasta_pipeline::finish ( size_t  lane_index,
std::span< gzip_file_probe const >  probes,
std::vector< inflated_genome > &  genomes,
std::vector< size_t > &  fallback 
)
inline

Waits for lane's batch and splits it into genomes the device holds and ids the host loader must take.

The genomes stay valid until release, which must follow once the kernels that read them are enqueued.

Definition at line 1259 of file device_gzip.cuh.

1264 {
1265 auto& lane = lanes_[lane_index];
1266 fallback.insert(fallback.end(), lane.fallback.begin(), lane.fallback.end());
1267 if (lane.files.empty()) return Ok();
1268 CUDDL_CUDA_TRY(lane.classified->sync());
1269 auto const table = host_table(lane);
1270 auto const overflow = *table.header_count > header_capacity_;
1271 auto const* bases = lane.decoded ? lane.slots->data() : lane.payload->data();
1272 uint64_t offset = 0;
1273 for (size_t file = 0; file < lane.files.size(); ++file) {
1274 auto const id = lane.files[file];
1275 auto const size = table.kept[file];
1276 bool intact = lane.decoded || probes[id].plain;
1277 if (!intact) {
1278 uint64_t inflated = 0;
1279 bool succeeded = true;
1280 for (auto m = lane.member_first[file]; m < lane.member_first[file + 1]; ++m) {
1281 succeeded = succeeded && table.member_statuses[m] == nvcompSuccess;
1282 inflated += table.member_inflated[m];
1283 }
1284 intact =
1285 succeeded && inflated == probes[id].isize && table.crcs[file] == probes[id].crc;
1286 }
1287 auto const fasta =
1288 table.first_char[file] != '@' && table.first_header[file] < table.slots[file + 1];
1289 // Each header contributes one separator, including the first header.
1290 // ponytail: CPU-validate shorter outputs; count per-file headers if this dominates.
1291 auto const has_sequence = size > *table.header_count;
1292 if (!overflow && intact && fasta && has_sequence) {
1293 genomes.push_back({id, bases + offset, size});
1294 } else {
1295 fallback.push_back(id);
1296 }
1297 offset += size;
1298 }
1299 if (static_cast<int64_t>(offset) != *table.selected) {
1300 return Err(Error::resource("device FASTA compaction lost track of its bytes"));
1301 }
1302 return Ok();
1303 }
#define CUDDL_CUDA_TRY(...)
Propagates a CUDA error wrapped in cuddl::Result<void>.
Definition error.hpp:248
@ fasta
FASTA (> headers).
Result< void > Ok() noexcept
Success return for Result<void>; same as return {}.
Definition error.hpp:181
cuda::std::unexpected< Error > Err(Error error)
Failure return; converts to any Result<T> via cuda::std::unexpected.
Definition error.hpp:176
static Error resource(std::string message)
Definition error.hpp:103
Here is the call graph for this function:

◆ hardware_limit()

size_t cuddl::detail::device_fasta_pipeline::hardware_limit ( ) const
inlinenoexcept

Largest inflated file the device's decompression engine takes; 0 without one.

Definition at line 812 of file device_gzip.cuh.

812 {
813 return hardware_limit_;
814 }

◆ lane_overhead()

static size_t cuddl::detail::device_fasta_pipeline::lane_overhead ( size_t  slot_capacity)
inlinestaticnoexcept

Device bytes a lane needs besides its slots and compressed bytes.

Definition at line 817 of file device_gzip.cuh.

817 {
818 return slot_capacity / 4096 * sizeof(uint64_t) + (size_t{64} << 20);
819 }

◆ release()

Result< void > cuddl::detail::device_fasta_pipeline::release ( size_t  lane_index,
cuda::stream_ref  consumer 
)
inline

Lets lane reuse its slots once consumer reaches this point.

Definition at line 1306 of file device_gzip.cuh.

1306 {
1307 CUDDL_CUDA_TRY(lanes_[lane_index].released->record(consumer));
1308 return Ok();
1309 }
Here is the call graph for this function:

◆ slot_capacity()

size_t cuddl::detail::device_fasta_pipeline::slot_capacity ( ) const
inlinenoexcept

Definition at line 803 of file device_gzip.cuh.

803 {
804 return slot_capacity_;
805 }

◆ submit()

Result< size_t > cuddl::detail::device_fasta_pipeline::submit ( size_t  lane_index,
std::span< size_t const >  ids,
std::span< std::filesystem::path const >  paths,
std::span< gzip_file_probe const >  probes,
bool  direct_reads 
)
inline

Starts inflating files from the front of ids on lane.

Every id must be a device candidate that fits one lane on its own. Plain files skip nvCOMP: they lead the lane's slots and reach them as read. Files with extra member signatures reach finish without device inflation.

Parameters
direct_readsreads the files with O_DIRECT, as read_whole_file does.
Returns
How many ids the batch took: batch_files of them.

Definition at line 848 of file device_gzip.cuh.

854 {
855 auto& lane = lanes_[lane_index];
856 CUDDL_TRY(cuda_try([&] { allocate(lane); }));
857 lane.decoded = false;
858 if (ids.empty()) return Err(Error::invalid_argument("an inflate batch needs a file"));
859 auto const files = batch_files(ids, probes);
860 // The pinned bytes are rewritten below, so the previous batch's upload must be done.
861 CUDDL_CUDA_TRY(lane.uploaded->sync());
862 auto table = host_table(lane);
863 // Plain files lie in the page-locked buffer exactly as they will in the slots, which
864 // they lead, so one copy places them all. Compressed bytes follow them.
865 uint64_t plain_at = 0;
866 for (size_t file = 0; file < files; ++file) {
867 auto const& probe = probes[ids[file]];
868 if (probe.plain) plain_at += probe.host_bytes();
869 }
870 uint64_t compressed_at = plain_at, slot_at = 0;
871 for (size_t file = 0; file < files; ++file) {
872 auto const& probe = probes[ids[file]];
873 table.compressed_bytes[file] = probe.compressed;
874 if (probe.plain) {
875 table.compressed_offsets[file] = slot_at + 1;
876 slot_at += probe.host_bytes();
877 } else {
878 table.compressed_offsets[file] = compressed_at;
879 compressed_at += probe.compressed;
880 }
881 }
882 std::atomic<bool> failed{false};
883 readers_.for_each(files, [&](size_t file) {
884 auto const* const path = paths[ids[file]].c_str();
885 auto* const out = lane.pinned->data() + table.compressed_offsets[file];
886 auto const bytes = table.compressed_bytes[file];
887 int const fd = ::open(path, O_RDONLY);
888 // Files lie back to back in the lane, so a direct read goes through a bounce buffer.
889 bool const read = fd != -1 && read_whole_file(fd, out, bytes, direct_reads);
890 auto const done = read ? bytes : 0;
891 if (!read) failed = true;
892 // A matching trailer cannot distinguish repeated first and last members.
893 // A signature inside compressed data is ambiguous too, so use the host parser.
894 // BGZF names every member in its block headers.
895 auto const& probe = probes[ids[file]];
896 lane.possible_members[file] =
897 !probe.plain && !probe.bgzf && done > 10 && has_gzip_signature(out + 10, done - 10);
898 });
899 if (failed) return Err(Error::invalid_argument("cannot read reference inputs"));
900 lane.files.clear();
901 lane.fallback.clear();
902 lane.member_first.clear();
903 engine_params_.clear();
904 slot_at = 0;
905 uint32_t members = 0;
906 auto const accept = [&](size_t file) {
907 table.slots[lane.files.size()] = slot_at;
908 lane.member_first.push_back(members);
909 slot_at += probes[ids[file]].isize + 1;
910 lane.files.push_back(ids[file]);
911 };
912 for (size_t file = 0; file < files; ++file) {
913 if (probes[ids[file]].plain) accept(file);
914 }
915 auto const plain = lane.files.size();
916 for (size_t file = 0; file < files; ++file) {
917 if (!probes[ids[file]].plain && lane.possible_members[file]) {
918 lane.fallback.push_back(ids[file]);
919 }
920 }
921 // One member: raw DEFLATE at @p deflate in the page-locked copy, which the payload buffer
922 // mirrors, inflating to @p size bytes at @p out.
923 auto const add_member = [&](unsigned char const* deflate,
924 uint64_t deflate_bytes,
925 uint32_t size,
926 char* out) {
927 auto* const source = lane.payload->data() +
928 (reinterpret_cast<char const*>(deflate) - lane.pinned->data());
929 table.member_sources[members] = source;
930 table.member_source_bytes[members] = deflate_bytes;
931 table.member_outputs[members] = out;
932 table.member_output_bytes[members] = size;
933 if (hardware_limit_ != 0 && size <= hardware_limit_) {
934 CUmemDecompressParams params{};
935 params.src = source;
936 params.srcNumBytes = deflate_bytes;
937 params.dst = out;
938 params.dstNumBytes = size;
939 // The low half of the member's 64-bit inflated size, zeroed before the batch.
940 params.dstActBytes =
941 reinterpret_cast<cuuint32_t*>(device_tables(lane).member_inflated + members);
942 params.algo = CU_MEM_DECOMPRESS_ALGORITHM_DEFLATE;
943 engine_params_.push_back(params);
944 }
945 ++members;
946 };
947 // Gzip files whose members all fit the decompression engine come first, then those
948 // nvCOMP inflates on the SMs, so each group's members form one range. Both take raw
949 // DEFLATE: the host steps over each member's header in its copy. Without an engine, every
950 // gzip file is in the second group.
951 auto const gzip_pass = [&](bool engine) {
952 for (size_t file = 0; file < files; ++file) {
953 auto const& probe = probes[ids[file]];
954 if (probe.plain || lane.possible_members[file]) continue;
955 bool const fits =
956 hardware_limit_ != 0 && (probe.bgzf || probe.isize <= hardware_limit_);
957 if (fits != engine) continue;
958 auto const* const bytes = reinterpret_cast<unsigned char const*>(
959 lane.pinned->data() + table.compressed_offsets[file]
960 );
961 auto* out = lane.slots->data() + slot_at + 1;
962 if (probe.bgzf) {
963 // The blocks must still be the ones probed, or they would overrun the slot.
964 auto const blocks = walk_bgzf(bytes, probe.compressed, [](auto...) {});
965 if (!blocks || blocks->first != probe.members ||
966 blocks->second != probe.isize) {
967 lane.fallback.push_back(ids[file]);
968 continue;
969 }
970 accept(file);
971 (void)walk_bgzf(
972 bytes, probe.compressed, [&](uint64_t at, uint64_t length, uint32_t size) {
973 add_member(bytes + at, length, size, out);
974 out += size;
975 }
976 );
977 continue;
978 }
979 auto const deflate_at = device_gzip::deflate_offset(
980 reinterpret_cast<char const*>(bytes), probe.compressed
981 );
982 if (deflate_at == 0) {
983 lane.fallback.push_back(ids[file]);
984 continue;
985 }
986 accept(file);
987 add_member(bytes + deflate_at, probe.compressed - deflate_at - 8, probe.isize, out);
988 }
989 };
990 gzip_pass(true);
991 auto const engine_members = members;
992 gzip_pass(false);
993 lane.member_first.push_back(members);
994 table.slots[lane.files.size()] = slot_at;
995 lane.slot_bytes = slot_at;
996 if (lane.files.empty()) return files;
997
998 auto const s = cuda::stream_ref{*lane.stream};
999 auto* const stream = s.get();
1000 auto const device_table = device_tables(lane);
1001 // The slots are rewritten below, so the sketch kernel reading them must be done.
1002 CUDDL_CUDA_TRY(s.wait(*lane.released));
1003 // Plain bytes land in their slots; slot_init_kernel then writes the guard bytes between.
1004 if (plain_at != 0) {
1006 cuda::copy_bytes(
1007 s,
1008 cuda::std::span{lane.pinned->data(), plain_at},
1009 device_span<char>{lane.slots->data(), plain_at}
1010 )
1011 );
1012 }
1013 if (compressed_at != plain_at) {
1015 cuda::copy_bytes(
1016 s,
1017 cuda::std::span{lane.pinned->data() + plain_at, compressed_at - plain_at},
1018 device_span<char>{lane.payload->data() + plain_at, compressed_at - plain_at}
1019 )
1020 );
1021 }
1023 cuda::copy_bytes(
1024 s,
1025 cuda::std::span{lane.host->data(), inputs_bytes()},
1026 device_span<char>{lane.tables->data(), inputs_bytes()}
1027 )
1028 );
1029 CUDDL_CUDA_TRY(lane.uploaded->record(s));
1030 auto const count = static_cast<uint32_t>(lane.files.size());
1031 device_gzip::slot_init_kernel<<<(count + 255) / 256, 256, 0, stream>>>(
1032 lane.slots->data(),
1033 device_table.slots,
1034 count,
1035 device_table.first_header,
1036 device_table.header_count
1037 );
1038 CUDDL_CUDA_TRY(cudaGetLastError());
1039 if (engine_members != 0) {
1040 // The driver takes the parameters from host memory. nvCOMP's own engine path builds
1041 // them on the device and synchronises the stream to read them back, which stalls this
1042 // thread and every lane it drives. The engine reports only inflated sizes, so the
1043 // statuses read as success; finish still checks each size and CRC32.
1044 static_assert(nvcompSuccess == 0);
1045 static_assert(std::endian::native == std::endian::little);
1046 CUDDL_CUDA_TRY(cudaMemsetAsync(
1047 device_table.member_inflated, 0, engine_members * sizeof(uint64_t), stream
1048 ));
1049 CUDDL_CUDA_TRY(cudaMemsetAsync(
1050 device_table.member_statuses, 0, engine_members * sizeof(nvcompStatus_t), stream
1051 ));
1052 size_t error_index = 0;
1053 auto const submitted = batch_decompress_(
1054 engine_params_.data(), engine_params_.size(), 0, &error_index, stream
1055 );
1056 if (submitted != CUDA_SUCCESS) {
1057 return Err(
1059 "hardware decompression launch failed: " + std::to_string(submitted)
1060 )
1061 );
1062 }
1063 }
1064 if (engine_members != members) {
1065 auto options = nvcompBatchedDeflateDecompressDefaultOpts;
1066 options.backend = NVCOMP_DECOMPRESS_BACKEND_CUDA;
1067 auto const first = engine_members;
1068 auto const inflated = nvcompBatchedDeflateDecompressAsync(
1069 reinterpret_cast<void const* const*>(device_table.member_sources + first),
1070 device_table.member_source_bytes + first,
1071 device_table.member_output_bytes + first,
1072 device_table.member_inflated + first,
1073 members - first,
1074 lane.temp->data(),
1075 nvcomp_temp_bytes_,
1076 reinterpret_cast<void* const*>(device_table.member_outputs + first),
1077 options,
1078 device_table.member_statuses + first,
1079 stream
1080 );
1081 if (inflated != nvcompSuccess) {
1082 return Err(
1083 Error::resource("nvCOMP DEFLATE launch failed: " + std::to_string(inflated))
1084 );
1085 }
1086 }
1087 if (auto const gzip = count - static_cast<uint32_t>(plain); gzip != 0) {
1088 device_gzip::slot_crc_kernel<<<gzip, device_gzip::block_size, 0, stream>>>(
1089 lane.slots->data(), device_table.slots + plain, device_table.crcs + plain
1090 );
1091 CUDDL_CUDA_TRY(cudaGetLastError());
1092 }
1093 CUDDL_TRY(normalise(lane, slot_at));
1094 return files;
1095 }
size_t batch_files(std::span< size_t const > ids, std::span< gzip_file_probe const > probes) const noexcept
How many ids from the front of ids one lane holds.
void for_each(size_t count, Work &&work)
Runs work(index) for every index below count; one call at a time.
#define CUDDL_TRY(expr)
Propagates a cuddl::Result failure from the enclosing function (GNU statement expression).
Definition error.hpp:225
size_t deflate_offset(char const *bytes, size_t size)
Where the DEFLATE stream starts in the size bytes of a gzip candidate (one member,...
auto const count
Definition kernels.cuh:1031
std::optional< std::pair< uint32_t, uint64_t > > walk_bgzf(unsigned char const *bytes, uint64_t size, Block &&block)
Calls block(deflate_at, deflate_bytes, inflated) for each non-empty block of the BGZF file in bytes.
Result< void > cuda_try(cudaError_t error, std::source_location location=std::source_location::current())
Checks a CUDA runtime call and returns an error on failure.
Definition error.hpp:187
static Error invalid_argument(std::string message)
Definition error.hpp:99
Here is the call graph for this function:

◆ submit_decoded()

Result< void > cuddl::detail::device_fasta_pipeline::submit_decoded ( size_t  lane_index,
std::span< decoded_fasta const >  sources 
)
inline

Copies CPU-inflated FASTA into a lane and enqueues GPU normalisation.

Retain the source bytes until this lane finishes, or until wait_lanes returns.

Definition at line 1181 of file device_gzip.cuh.

1181 {
1182 if (sources.empty() || sources.size() > max_files || compressed_capacity_ != 0) {
1183 return Err(Error::invalid_argument("invalid decoded FASTA batch"));
1184 }
1185 auto& lane = lanes_[lane_index];
1186 CUDDL_TRY(cuda_try([&] { allocate(lane); }));
1187 lane.uploaded->sync();
1188 lane.decoded = true;
1189 lane.files.clear();
1190 lane.fallback.clear();
1191 auto const table = host_table(lane);
1192 uint64_t bytes = 0;
1193 for (auto const& source : sources) {
1194 if (source.bytes.size() >= slot_capacity_ - bytes) {
1195 return Err(Error::invalid_argument("decoded FASTA batch exceeds its lane"));
1196 }
1197 auto const file = lane.files.size();
1198 table.slots[file] = bytes;
1199 table.compressed_ptrs[file] = const_cast<char*>(source.bytes.data());
1200 table.compressed_bytes[file] = source.bytes.size();
1201 table.output_ptrs[file] = lane.slots->data() + bytes + 1;
1202 bytes += source.bytes.size() + 1;
1203 lane.files.push_back(source.id);
1204 }
1205 table.slots[sources.size()] = bytes;
1206 auto const s = cuda::stream_ref{*lane.stream};
1207 auto const device_table = device_tables(lane);
1208 CUDDL_CUDA_TRY(s.wait(*lane.released));
1210 cuda::copy_bytes(
1211 s,
1212 cuda::std::span{lane.host->data(), inputs_bytes()},
1213 device_span<char>{lane.tables->data(), inputs_bytes()}
1214 )
1215 );
1216 auto const count = static_cast<uint32_t>(sources.size());
1217 device_gzip::slot_init_kernel<<<(count + 255) / 256, 256, 0, s.get()>>>(
1218 lane.slots->data(),
1219 device_table.slots,
1220 count,
1221 device_table.first_header,
1222 device_table.header_count
1223 );
1224 CUDDL_CUDA_TRY(cudaGetLastError());
1225 static_assert(CUDART_VERSION >= 13000, "cuDDL needs CUDA 13.0 or newer");
1226 cudaMemcpyAttributes attributes{};
1227 attributes.srcAccessOrder = cudaMemcpySrcAccessOrderAny;
1228 attributes.srcLocHint = {cudaMemLocationTypeHost, 0};
1229 attributes.flags = cudaMemcpyFlagPreferOverlapWithCompute;
1230 size_t attributes_index = 0;
1231 CUDDL_CUDA_TRY(cudaMemcpyBatchAsync(
1232 reinterpret_cast<void* const*>(table.output_ptrs),
1233 reinterpret_cast<void const* const*>(table.compressed_ptrs),
1234 table.compressed_bytes,
1235 count,
1236 &attributes,
1237 &attributes_index,
1238 1,
1239 s.get()
1240 ));
1241 CUDDL_CUDA_TRY(lane.uploaded->record(s));
1242 return normalise(lane, bytes);
1243 }
Here is the call graph for this function:

◆ wait_lanes()

Result< void > cuddl::detail::device_fasta_pipeline::wait_lanes ( cuda::stream_ref  consumer)
inline

Completes lane work before callers release outstanding input buffers.

Definition at line 1246 of file device_gzip.cuh.

1246 {
1247 for (auto& lane : lanes_) {
1248 CUDDL_CUDA_TRY(lane.stream->sync());
1249 }
1250 CUDDL_CUDA_TRY(consumer.sync());
1251 return Ok();
1252 }
Here is the call graph for this function:

Member Data Documentation

◆ lane_count

constexpr size_t cuddl::detail::device_fasta_pipeline::lane_count = 6
staticconstexpr

Definition at line 715 of file device_gzip.cuh.

◆ max_files

constexpr size_t cuddl::detail::device_fasta_pipeline::max_files = 4096
staticconstexpr

Definition at line 716 of file device_gzip.cuh.

◆ max_members

constexpr size_t cuddl::detail::device_fasta_pipeline::max_members = 65536
staticconstexpr

Gzip members one batch inflates: one per file, or one per non-empty BGZF block.

With BGZF's 64 KiB blocks, about 4 GiB of output.

Definition at line 719 of file device_gzip.cuh.


The documentation for this class was generated from the following file: