Changes in version 0.7.0 Features - bixverse-rs 0.7.0 under the hood, with faer 0.24 and edge-rs 0.3. - Leiden and Louvain now run in Rust: leiden_clustering() on any igraph, and find_clusters_sc() uses it. Results depend on seed only, not on the global RNG or the thread count. - params_sc_fast_cluster() gained cell_mediated and n_anchors for the cell-mediated centroid graph from Secuer (Wei et al., PLoS Comput Biol, 2022). - calc_meta_cell_compactness() and calc_meta_cell_separation() score any meta cell assignment against a diffusion map, including meta cells built outside bixverse. Performance - Small cell selections (up to 10% of the store and 100k cells, i.e. most SingleCellsSubset objects) are read from the cell-major file for HVG detection and the residual fits. HVG on 6k of 985k cells: 0.82 s to 43 ms. - Fused sparse batch readers on the Rust side; SCENIC, Scrublet, NEBULA, CellPhoneDB, NMF, PCA, CellSweep, Bonsai and meta cells build their matrices through them. - Exact covariance PCA uses the cell Gram when there are fewer cells than genes, so small subsets no longer pay for a gene x gene eigendecomposition. - Subset filters are pushed into DuckDB, and meta cell matrices are built without copies. Changes - Clustering results shift compared to 0.6.x, since the igraph Leiden/Louvain calls are gone. Fixes - set_residual_fit(), get_residual_fit() and remove_residual_fit() work on SingleCellsSubset. - Accelerate is linked properly on macOS. Changes in version 0.6.3 Fix - Make the compile work on Windows arm64 Changes in version 0.6.2 Features - Wired in faster NEBULA via edge-rs. Fix - Fixed the bug that h5ad ingest silently drops string and nullable obs columns from the obs table. - Personalised PageRank stopped iterating too early; it now matches igraph (default tol = 1e-12, rs_page_rank* gained damping_factor and tol) (#254). Changes in version 0.6.1 Features - save_h5ad() exports a SingleCells object to an AnnData .h5ad file for ScanPy: raw or normalised counts in X, obs/var, and cached PCA, embeddings and sNN graph (obsm/varm/obsp). Counts are streamed in cell batches and written via scx-core. - as_ligand_target_influence() wraps a ligand-target matrix built elsewhere (e.g. nichenetr) as a LigandTargetInfluence, so it can be scored with ligand_activity_scores(). Changes in version 0.6.0 Larger release with performance improvements across the board and new methods. The Rust backend got a sweep and parts that could be parallelised, but were not are now parallel, SIMD/GEMM integrated everywhere applicable and better memory access patterns across the board. Some of the rounding might be slightly different, hence, major release. Features - CellPhoneDB wired in, see cellphonedb_sc() and get_cellphonedb_db(). The NicheNet vignette is now a general ligand-receptor vignette. - scran/scrapper-style HVG detection via params_sc_hvg(method = "scran"). - svd_solver for the PCA in the doublet detection methods. - The solver for Harmony version 1 was updated and is substantially faster. - Option to archive the binary files now via archive_sc_exp(). The gene-centric view gets deleted and the cell-centric view gets massively compressed. Ideal for archiving after large analysis are done. - The sNN graph re-attaches small isolated islands after pruning in single cell again. - Changes to the ICA QC metrics. Instead of mean mutual information, it returns now 1 - max(abs(cor)) to identify the point where two ICs are strongly correlated with each other. - The randomised SVD paths for single cell are now using N_OVERSAMPLING from 100 to 20 and increase the iterations from 2 to 4. Across various benchmarks and synthetic data sets, the quality of the trailing PCs has improved with the new thresholds while being faster. - NEBULA has now a parameter that enables that a minimum of samples need to have the minimum amount of cells expressing a gene in terms of proportion and min expression. Fixes - hvg_method = "mvb" in the doublet params silently fell back to VST. The choice is now "meanvarbin", plus the new "scran". Changes in version 0.5.7 Features - Dramatically improved speed for find_all_markers_sc() and find_specific_markers_sc(). - Added extract_milo_plot_data() to generate plots in conjuction with bixverse.plots≥0.2.9. Fixes - Added reverse edges to the miloR kNN graph which yields larger neighbourhoods and better statistics. This changes results to prior versions. Changes in version 0.5.6 Features - Accelerate for MacOS users properly wired in. Changes in version 0.5.5 Features - Bonsai and Sanity wired in via their respective Rust implementations, bonsai-rs and sanity-sc-rs. - Helper function to transform TF on/off matrices into heatmap inputs, see extract_binary_heatmap_data(). Fixes - Full access to all of the kNN backends in manifoldsR from the single cell-related 2D embedding methods. Changes in version 0.5.4 Features - New single cell I/O machinery from bixverse-rs 0.5.6. The cell-to-gene (CSR to CSC) conversion now has one knob, csc_mem_gb: NULL (default) converts in one pass, a number caps the conversion buffers at that many GB. Available on all count loaders (load_mtx(), load_multi_mtx(), load_h5ad(), load_h5ad_norm(), load_multi_h5ad(), stream_h5ad(), load_tenx_h5(), load_multi_tenx_h5(), load_seurat(), load_sce(), load_r_data()), plus merge_sc_experiments() and cellsweep_sc(). - load_h5ad() gained h5ad_streaming (default TRUE), which streams the counts into the cell-based binary in batches instead of materialising the filtered matrix first. stream_h5ad() is now an alias for load_h5ad(h5ad_streaming = TRUE). - More 2D embeddings from manifoldsR 0.3.5: densmap_sc() and densne_sc() for the density-preserving versions of UMAP and t-SNE (Narayan et al.), and forceatlas2_sc() for ForceAtlas2 layouts of either the kNN or the sNN graph. See the updated plotting vignette. - tsne_sc() and densne_sc() take approx_type = "fft_3k", the three-kernel FFT optimiser (Unix only, like "fft"). Changes - streaming, batch_size, max_genes_in_memory and cell_batch_size are deprecated on every function listed above. They are ignored and warn if supplied; use csc_mem_gb instead. Nothing breaks, old calls keep running. - Needs manifoldsR >= 0.3.5. Fixes - h5ad: the number of cells and genes is read from the _index of /obs and /var rather than the first dataset in the group, and CSR vs CSC comes from the encoding-type attribute of X instead of being guessed from the indptr length. - tsne_sc() documentation had the use_knn default and the perplexity default wrong. Changes in version 0.5.3 Features - Single cell PCA got a lot faster. params_sc_pca() has a new svd_solver argument, one of "covariance" (the new default), "randomised" or "exact". The covariance solver builds the gene x gene cross-product straight from the sparse counts and eigendecomposes it. The randomised solver got about 2x faster as well. - calculate_pca_sc() and step_pca_sc() now default to sparse_svd = NULL, which picks the sparse path for the normalised data (faster and lighter on memory for every solver) and the dense one for residuals = TRUE. - PCA on matrices with missing values: run_ppca() and run_bpca() for probabilistic and Bayesian PCA, ported from pcaMethods. Changes - The default PCA is now exact rather than randomised, so PCs, and everything built on them, shift slightly compared to 0.5.2. The leading PCs barely move; the trailing ones can. - The randomised argument of params_sc_pca() is deprecated. TRUE maps to svd_solver = "randomised", FALSE to "exact", with a warning. Fixes - Cells that the sNN pruning left without a single edge now keep their strongest kNN edge. Before, Leiden turned each of them into a one-cell cluster, and on large data these added up to hundreds. - The sparse PCA path ignored mean_center = FALSE and normalise_variance = FALSE. Changes in version 0.5.2 Features - Cellsweep implementation from Caskey et al., see vignette. - Wired in new single cell batch correction metrics (cell-type specific LISI and ASW). - Also wired in from edge-rs the limma-voom workflow. - Sweep over the Rust/rextendr documentation - Wired in scTransform v2 from Choudhary and the vignette. Fixes - Good old 0 to 1-index indexing issues fixed in two functions. Changes in version 0.5.1 Documentations - Large sweep across all documentations to add examples, remove staleness and harmonise spelling. Fixes - Some minor bug fixes that became apparent during the doc sweep. Changes in version 0.5.0 Features - blitzGSEA wired into the R layer via calc_blitzgsea(). Permutation GSEA pays per pathway, this does not: random gene sets are drawn once across a log-spaced grid of set sizes, gamma tails are fitted to the null enrichment scores, and every pathway's p-value is read straight off the fitted tail. One gamma evaluation per pathway, no matter how big the library. No gene set library enters the calibration, so blitzgsea_calibrate() gives you a BlitzGseaNull that serves every library scored against that signature and survives saveRDS(). params_blitzgsea() and a print() method come with it. - bixverse-rs 0.5.0 under the hood. kNN graphs now return true Euclidean distances, so the is_squared plumbing is gone from the binding layer. - NNDescent gained extract_knn: hand back the graph the descent already built instead of running a separate search over the index. - load_sce() reads SingleCellExperiment objects. The Seurat loader was deduplicated against it. - Windows builds work on the r-universe runners. Fixes - plot.Hotspot() draws readable module blocks, and the scored modules are the ones the heatmap actually shows. - ICA whitening rank is capped rather than left to blow up on rank-deficient input. - The external HDF5 is gated on a cross-ABI host. Changes in version 0.4.11 Features Single cells - NEBULA for mixed model differential expression, see He, et al., Commun Biol, 2021: nebula_sc() on SingleCells and nebula_mc() on MetaCells, for when you have repeated measures per subject and pseudobulk throws away too much. - The miloR neighbourhood test runs in Rust now via rs_make_milor_nhoods(), behind get_miloR_abundances_sc() and test_nhoods(). - edgeR quasi-likelihood workflow via the edge-rs crate: run_edger_ql() does the fit in Rust rather than round tripping through the R package. - Fewer redundant copies in the meta cell sparse handoff. Chores - Linux binary builds, and vignettes are kept out of the build tarball. - Windows builds happen outside the package directory to dodge MAX_PATH. - r-universe is now the easy install route. Changes in version 0.4.10 Features - Consensus NMF implemented: - consensus_nmf_bulk() on BulkCoExp and consensus_nmf_sc() on SingleCells and MetaCells. - nmf_k_sweep_bulk() and nmf_k_sweep_sc() sweep a range of k and report stability against reconstruction error, so choosing k stops being a guess. Both return an NmfKSweepResult with a plot() method. - params_nmf_consensus() plus the matching checkmate assertions. - New ConsensusNmfResult class with get_w(), get_h(), get_data() and a new get_stability() generic for the clustering diagnostics. - get_nmf_stability() on BulkCoExp now covers consensus fits as well as stabilised ones. - Agent skills implemented: install_agent_skill() installs an agent skill. - VISION scores enabled also for the MetaCells class. - Latent dirichlet allocation (LDA) implemented for decomposing binary data (think cell x TF active binary matrices). - DIALOGUE also available now. Changes in version 0.4.9 Chores - Update to the test suite to have less redundancies, be cleaner. - Version bumps on the Rust side. - Simplified testing suite with less redundancies. Fixes - Added @importFrom Matrix sparseMatrix to the package import to avoid issues with meta cells and the conversions of the sparse matrices to lists for Rust. Changes in version 0.4.8 Features Single cells Meta cells: - Improved speed for meta cell methods regarding HVG detection and PCA. - Additional functionalities around meta cell purities: you can return entropy top label and second best label + proportions. - Hotspot can now also be applied on meta cells. Single cells: - Regulon binarisation for SCENIC. - AUCell now defaults to "recovery" a more top heavy score. If you want the old behaviour, please use "wilcox". - One-vs-many comparisons to identify cell type markers. Fixes - Various edge cases bug fixes in bixverse-rs that propagate into here. Improved dev experience If you want to develop on the package, set: Sys.setenv(DEV_BUILD = "true") devtools::document() devtools::install() for local testing and exploration. It will optimise less aggressively in terms of performance, but still reach very good speeds this way, while massively reducing compile times. Changes in version 0.4.7 Features Single cells - Faster HVG detection across the board (batch-aware, streaming, one-pass) with faster Loess that uses interpolation now. Results compared to prior runs for "vst" might be slightly different now. - Implementation of two trajectory methods: PAGA and Palantir. - State management added for all of the single cell parts in the ScCache part. Less foot guns in the future. This works still with older objects, but good practice for the future. - Improvements on the meta-cells: - cells_to_keep are stored now avoiding some problems with state. - Less memory-hungry merging implemented. Changes in version 0.4.6 Features Single cells - bixverse-rs version bump for better performance in SEACells and more robust streaming engine. Old files will still behave, but the desired target size is now encoded in the files. - The pruning operation has been set to TRUE by default in SEACells after further benchmarking. - Updated vignettes that had stale information (especially meta cells). Documentation - Missing documentation fixes for parameters. Changes in version 0.4.5 Features Single cells - Further improvements to the pipelines with meta cells enabled. - Merging of meta cells together. Changes in version 0.4.4 Features General - NMF implemented for bulk co-expression module detection and updated vignette. - Synthetic data generators for bulk updated to create different data structures. Single cells - Improved cell quality control functions. - Additional single cell batch correction methods: - Seurat-inspired CCA anchor-based batch correction - Seurat-inspired rPCA anchor-based batch correction - SingleCellsSubset class. This class creates a view into a subset of cells, think doing further analysis on T cells within your data set. - AUCell reworked. aucell_sc() now takes a params_sc_aucell() list instead of an auc_type string, and offers three statistics: "wilcox" (the Mann-Whitney AUC, still the default), "recovery" (the recovery-curve AUC of Aibar et al., with a max_rank cutoff) and "ap" (average precision). Scores can be z-scored per gene set via standardise. - Something is cooking for the single cell stuff. There has been a major refactor on the Rust side that will enable some very, very, very cool future features. Watch the space... Breaking changes - aucell_sc(auc_type = ...) is gone, use aucell_sc(aucell_params = params_sc_aucell(auc_type = ...)). The "auroc" value has been dropped and now errors. "wilcox" results are unchanged. Changes in version 0.4.3 Fix - Bumped version of bixverse-rs that has a fix for the IVF-type approximate nearest neighbour seraches. Changes in version 0.4.2 Features - Faster fgsea multi-level implementation with less unnecessary allocations and faster computations. Fix - Edge case in fgsea multi-level fixed that broke the beta-value calculations. Breaking changes get_centroids() was renamed to get_centroids_sc() for the usage on the fast clustering algorithms. Changes in version 0.4.1 Features - SingleCellsSubset class that can split from the main class and allows for analysis of e.g., a subset of cell types, etc. - Added singscore, see Foroutan et al., as another option to score single samples and their pathway activity. Changes in version 0.4.0 Features Following things were added: - Merging of SingleCells object possible - Multi file reading in for .mtx files - Possibility to read in densely stored h5ad files. - 10x h5 files can be read in. - Multi-model single cell support set up - New multi-model class for single cells: SingleCellsMultiModal. - You can no add the ADT counts to that one (in the future maybe also ATAC- Seq). - The weighted nearest neighbour graph method from Hao et al., 2021 - Improved performance on: - Harmony version 1 and version 2 with substantial speed improvements. - Hotspot with even more substantial speed improvements via doing less suboptimal things in the Rust code. - Large improvements to the memory management of SuperCells and SEACells. Both can run on a million cells on consumer hardware now. - Large number of helpers for plotting via the sister package and updated vignettes for this. - Reading in of 10x CellRanger h5 files (and multi-file support). - NMF for single cell wired up. TODO: Add also to bulk -> in one of the future releases. - Symphony for label transfer of cell types. - Grouping column for doublet detection methods. - NicheNet implementation for ligand receptor analysis Bug fixes - Multi-file loaders (load_multi_tenx_h5(), load_multi_h5ad()) no longer write NA gene ids into the var table under gene_universe = "union" when the var reference file does not contain every gene in the universe. The gene_id is now pinned to the canonical universe id and additional data found in the vars is dropped. The user has to add them subsequently. - Properly dealt with the mitch bug and removed the hotfix part. Breaking changes The interface to the i/o functions for the single cell was changed. The streaming = TRUE parameter was simplified and gives you now more control over how to stream in data. This can (and will) break old code! If you get an error like: Assertion on 'streaming' failed. Must be of class 'integer', not 'logical'. Update the streaming parameter to 0L, 1L or 2L, pending on data set size and available memory. Additionally, the interface to the PCA function has changed, see here: calculate_pca_sc( object, no_pcs, pca_params = params_sc_pca(), # new parameter replacing randomised_svd sparse_svd = FALSE, hvg = NULL, seed = 42L, .verbose = TRUE ) This gives you way more control over how to run the PCA normalisation (mean centering, variance normalisation or now also the new PFLogPF normalisation). Changes in version 0.3.3 Fixes - Hotfix coded in that made mitch explode on larger data sets. Will be properly fixed on the bixverse-rs side in the next release, but needed now. Changes in version 0.3.2 Features Following things were added: - scDblFinder as another doublet detection method - Following methods on top of meta-cells: - HVG detection - PCA - kNN generation - sNN generation - Leiden clustering - SCENIC for meta cells - Exposing (finally) other HVG methods for single cell - Addition of KmKnn as a nearest neighbour method - Fast clustering methods with k-means clustering followed by Louvain clustering. - Harmony v2 added. - Following vignettes added: - K-nearest neighbour searches for single cell - Metacells in bixverse - Updates to various single cell vignettes - Major update to the rextendr/extendr interface Breaking changes Changes in version 0.3.1 - Minor dependency update on the Rust side. Bumped versions to improved Rust crates. Changes in version 0.3.0 The big one... First, shift of versioning + release of full single cell analysis suite. See below for more details. - Full refactor of the Rust code, so it lives in its own crate - Single cell officially released: - Class with getters that leverages on-disk storage of counts and a streaming engine to enable memory-friendly analysis of millions of cells. - I/O helpers that can read mtx outputs, h5ad, h5ad with only normalised counts, multiple h5ad and conversion from Seurat implemented. - Fully functional suite of pre-processing functions: proportion of gene sets, proportion of Top N genes of overall counts, HVG detection, PCA (with sparse solving avoiding densification), multiple kNN backends for different use-cases, sNN generation and Leiden community detection. - Three different batch correction methods implemented: BBKNN, fastMNN and Harmony. - Various analysis methods for single cell implemented: AUCell, GeneModuleScoring, VISION, Hotspot, SCENIC and miloR implementations. - Metacell generation: hdWGCNA bootstrap approach, SuperCells and SEACells. - Large number of convenience functions. - Sister package started with plotting functions. - Package with very fast 2D embedding methods also ready as a sister package – check out manifoldsR. Changes in version 0.0.2.3 - RCisTarget implemented in Rust, see Aebir et al. for details. - Bug fix: scaling in preprocess_bulk_coexp() caused a matrix transpose. Changes in version 0.0.2.2 - Mitch multi contrast enrichment implemented in Rust. - Rework and refactor of sub modules in the Rust code. - Additional Rust functions for set similarity across a list of items in R. - QC function for the bulk DGE class split into qc_bulk_dge() and normalise_bulk_dge(). - Bulk DGE function split into calculate_dge_limma() and calculate_dge_hedges(). - Splitting by correlation sign and Eigengene calculation added for CoReMo. - SNF algorithm added from Wang et al.. - Spectral clustering methods implemented in Rust. - Breaking change: rs_set_similarity_list() renamed to rs_set_similarity_list2(). The former now takes a single list and calculates set similarity across all permutations of elements. - Breaking change: preprocess_bulk_dge() and calculate_all_dges() have been removed. Changes in version 0.0.2.1 - Constrained PageRank implementation added. - Improved synthetic data for correlation-based gene module detection. - Improved tests for ICA, CoReMo and graph-based correlation modules. - Various distance calculations accelerated in Rust. - Rust-based personalised PageRank implementations now support weighted graphs. - Bug fixes for the graph-based correlation methods. - Fixed export bugs for some plotting functions. - Breaking change: interface to Rust-based personalised PageRank calculations may break due to the addition of a weights parameter. Changes in version 0.0.2.0 - Rework of the ontology class with Wang similarity added as an additional measure. - Rework of the genetic community detection class and diffusion methods, including permutation-based testing for diffusions. - Rework of the ICA code with a testing suite added for the ICA class. - Reciprocal best hit method based on correlations added; an additional parameter is now required specifying whether to use set similarity or correlation-based similarity. - Vignettes added for various methods. - GSVA and ssGSEA implemented in Rust. - Rust code reworked in various places for improved performance. - Mutual information calculations implemented in Rust. - Improvements to the synthetic data. - Bug fixes for the graph-based correlation methods. - Breaking change: ontology class functions and methods have been modified heavily and may be breaking. - Breaking change: community detection method has been updated; old code may not work. - Breaking change: ICA detection functions and methods have been updated and may break existing code. Changes in version 0.0.1.4 - TOM implemented in Rust for correlation-based module detection. - CoReMo-based gene module detection added. - Breaking change: cor_module_check_res() renamed to cor_module_graph_check_res() and cor_module_final_modules() renamed to cor_module_graph_final_modules(). Changes in version 0.0.1.3 - fgsea multi level method implemented; n_more_extreme is now returned by GSEA functions. - RBF function implementations added for R matrices. Changes in version 0.0.1.2 - fgsea simple implemented for gene ontology with elimination method. - Improved gene ontology elimination methods to reduce unnecessary copying. Changes in version 0.0.1.1 - Semantic similarities for ontologies added. - Speed improvements in various Rust functions via reduced memory copying and use of lifetimes. - Package stability improved with tests powered by tinytest. - Set similarity Rust functions exposed. - DGE class added for Limma Voom differential gene expression calculations. - Wrapper functions added to load h5ad objects into R memory. - Traditional GSEA and fgsea simple ported into Rust. - Bug fix: future::plan() for iterating over resolutions in reciprocal best hit graph generation. - Bug fix: hypergeometric calculations and RBH graph. - Breaking change: community_detection() now uses params_community_detection() for parameters. Changes in version 0.0.1.0 - Hypergeometric tests for gene set analysis for single or lists of target gene sets. - Gene ontology aware hypergeometric test for GO analysis. - Network diffusion methods based on personalised PageRank (single and tied diffusion). - Community detection algorithms on top of diffusion scores. - Reciprocal best hit graphs using set similarities between gene modules from different data sets or methods. - Contrastive PCA implementation for gene module detection. - Differential correlation-based methods for gene module detection using network-based graph community detection. - Independent component analysis with stabilised versions. - Helper functions for Hedge's G effect size and OpenTargets score summarisation.